Server, remote control system, remote control method, and program

Abstract

To make a robot appropriately operate.SOLUTION: A server includes: a communication part communicating with one or more robot devices and one or more operation terminals remote-controlling at least one of the one or more robot devices via network; an acquisition part acquiring operation contents input by the operation terminal; an intention estimation part estimating intention of an operator at the operation terminal on the basis of the operation contents acquired by the acquisition part and also updating intention estimation information; and an operation control part generating operation control information for operating the target robot device on the basis of an intention result estimated by the intention estimation part and the updated intention estimation information, and controlling the operation of the robot device on the basis of the generated operation control information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a server, a remote control system, a remote control method, and a program. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known a technique for remotely controlling a robot to operate laboratory equipment or laboratory devices using data transmitted from a monitor terminal or another terminal (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-092749 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, there are limitations on the instructions that can be given from a terminal device, so the robot can only be made to perform specific tasks, and there are cases where the robot cannot be made to operate appropriately.

[0005] The aspects of the present invention have been made in consideration of these circumstances, and one of their objectives is to provide a server, a remote control system, a remote control method, and a program that can operate a robot more appropriately. [Means for solving the problem]

[0006] The server, remote control system, remote control method, and program according to the present invention employ the following configuration. (1): A server according to one aspect of the present invention includes a communication unit that communicates via a network with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices; an acquisition unit that acquires operation content input by the operation terminal; an intention estimation unit that estimates the intention of an operator of the operation terminal based on the operation content acquired by the acquisition unit and updates intention estimation information; and an operation control unit that generates operation control information for operating a target robotic device based on the intention result estimated by the intention estimation unit and the updated intention estimation information, and controls the operation of the robotic device based on the generated operation control information, wherein the intention includes the intention of an action that the operator wants the robotic device to perform and a prediction of the action, and the intention estimation information is a learning model trained using training data for inputting operation content for the robotic device and outputting correct intention information for the operation content, and the intention estimation unit updates the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robotic device.

[0007] (2): In the above aspect (1), the acquisition unit acquires robot control information relating to the operation of the robot device and sensor information detected by a sensor provided in the robot device, and further includes a management unit that sets different access rights for the robot control information and the sensor information acquired by the acquisition unit, and enables acquisition of either the robot control information or the sensor information when the set access rights are confirmed.

[0008] (3): In the above-mentioned aspect (1) or (2), when the operation control unit controls the operation of one robot device based on operation contents acquired from multiple operation terminals, the operation control information is generated based on the priority of the operator operating the operation terminal.

[0009] (4): In the aspect (3) above, the intention estimation unit further includes an output control unit that estimates the intention of each operator of a plurality of operation terminals that operate the single robot device, and outputs the intention estimation results of each operator estimated by the intention estimation unit to each of the plurality of operation terminals.

[0010] (5): Another aspect of the present invention is a remote control system comprising the server described in any one of (1) to (4) above, the one or more robot devices, and the one or more operation terminals.

[0011] (6): Another aspect of the present invention is a remote operation method in which a computer communicates via a network with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices, acquires operation content input by the operation terminal, estimates the intention of an operator of the operation terminal based on the acquired operation content, and updates intention estimation information, generates operation control information for operating a target robotic device based on the estimated intention result and the updated intention estimation information, and controls the operation of the robotic device based on the generated operation control information, the intention including the intention of an action that the operator wants the robotic device to perform and a prediction of the action, the intention estimation information is a learning model trained using training data for inputting operation content for the robotic device and outputting correct intention information for the operation content, and updates the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robotic device.

[0012] (7): Another aspect of the present invention is a program that causes a computer to communicate via a network with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices, acquire operation content input by the operation terminal, estimate the intention of an operator of the operation terminal based on the acquired operation content, and update intention estimation information, generate operation control information for operating a target robotic device based on the estimated intention result and the updated intention estimation information, and control the operation of the robotic device based on the generated operation control information, wherein the intention includes an intention of an action that the operator wants the robotic device to perform and a prediction of the action, and the intention estimation information is a learning model trained using training data for inputting operation content for the robotic device and outputting correct intention information for the operation content, and updates the intention estimation information based on history information that indicates the operation content input by the operator and the actual operation content of the robotic device. [Effects of the Invention]

[0013] According to the above aspects (1) to (7), the robot can be operated more appropriately.

[0014] According to the above aspect (2), data confidentiality is protected. Therefore, for example, it is possible to prevent sensor information from being stolen or tampered with by companies that do not have access rights, thereby improving the security of the entire system. Furthermore, by managing the information provided based on access rights, various companies can use the service with peace of mind.

[0015] According to the above aspect (3), even when the robot is operated by a plurality of operators, the robot can be operated appropriately. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of a configuration of a remote control system 1 according to an embodiment. [Figure 2] 1 is a diagram illustrating an example of a configuration of a management server 100 according to an embodiment. [Figure 3] FIG. 10 is a diagram for explaining the contents of intention estimation information 172B. [Figure 4] FIG. 10 is a diagram for explaining the contents of company-specific user information 174A. [Figure 5] FIG. 10 is a diagram for explaining the contents of company-specific sensor information 174B. [Figure 6] FIG. 1 is a diagram illustrating an example of a configuration of a robot device 200 according to an embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a configuration of an operation terminal 300 according to the embodiment. [Figure 8] FIG. 2 is a diagram for explaining remote control of the robot device 200 according to the present embodiment. [Figure 9] 10 is a diagram showing an example of an image IM10 displayed by the operation terminal 300. FIG. [Figure 10] FIG. 10 is a diagram showing an example of an image IM20 displayed on the operation terminal 300 when a task based on a priority is executed. [Figure 11] FIG. 10 is a diagram showing an example of an image IM30 that inquires of an operator about an operation to be executed by the robot device. [Figure 12] FIG. 10 is a diagram showing an example of an image IM40 displayed on a monitor 214 of the robot device 200. [Figure 13] FIG. 2 is a sequence diagram illustrating a process executed by the remote control system 1 according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of a server, a remote control system, a remote control method, and a program according to the present invention will be described with reference to the drawings.

[0018] [System Configuration] FIG. 1 is a diagram illustrating an example of the configuration of a remote operation system 1 according to an embodiment. The remote operation system 1 illustrated in FIG. 1 includes, for example, a management server 100, a robot device 200, and an operation terminal 300. In addition to the above configuration, the management server 100 may also include a robot manufacturer terminal 400 and a telecommunications company terminal 500. The management server 100 is communicatively connected to the robot device 200, the operation terminal 300, the robot manufacturer terminal 400, and the telecommunications company terminal 500 via a network NW. The network NW includes, for example, the Internet, a cellular network, a Wi-Fi (registered trademark) network, a wide area network (WAN), a local area network (LAN), a provider device, a wireless base station, etc. The remote operation system 1 may include one or more robot devices 200 and one or more operation terminals 300. While the example in FIG. 1 illustrates robot devices 200A, 200B, and 200C, the number and types of robot devices are not limited thereto. 1 shows operation terminals 300A-1 to 300A-n (n is a natural number equal to or greater than 2) and 300B-1 to 300B-n managed by each of a plurality of companies A and B, but the number of such terminals is not limited to this. Hereinafter, when the robot device and the operation terminal are not particularly identified, they will be simply referred to as "robot device 200" and "operation terminal 300." The management server 100 is an example of a "server."

[0019] The management server 100 manages one or more operation terminals 300 in association with one or more robotic devices 200 to be operated. In this case, the management server 100 may associate one robotic device 200 with each operation terminal 300, may associate one robotic device 200 with multiple operation terminals 300, or may associate multiple robotic devices 200 with one operation terminal 300. The management server 100 also acquires operation details input from the operation terminal 300 and transmits operation control information corresponding to the operation details to the robotic device 200, thereby remotely operating the robotic device 200. The management server 100 also transmits information acquired by the robotic device 200 to the operation terminal 300.

[0020] The robotic device 200 performs a predetermined operation based on control information transmitted from the management server 100 via the network NW. The robotic device 200 includes a mobile mechanism that can move by driving, for example, wheels, a cart, a crawler, a crane, or the like. In the example of FIG. 1, the robotic device 200A is a robot equipped with both arms that can move using at least a crane. The robotic device 200B is a robot equipped with both arms that can move using wheels attached to the bottom. The robotic device 200C is a robot equipped with both arms that can move by walking on two legs. The robotic device 200 also includes an arm unit that performs tasks such as grasping, moving, and manipulating an object. The robotic device 200 is also equipped with multiple sensors, such as vibration, temperature, pressure, and tactile sensors, and transmits data detected by each sensor to the management server 100 at a predetermined interval or when a request is received. The robotic device 200 may also include a camera that captures images of the surrounding area, a monitor that displays images, a speaker that outputs audio, a microphone that captures surrounding sounds, and the like.

[0021] For example, one or more operation terminals 300 are provided for each company that uses the robotic device 200. The operation terminal 300 inputs operation details for the robotic device 200 and notifies the operator of information acquired by the robotic device 200. The operation terminal 300 can operate one or more robotic devices 200. The operation terminal 300 may also operate some parts of the robotic device 200 (e.g., the right arm or the left arm). The operation terminal 300 may also be provided with, for example, an HMD (head-mounted display), an operation device, and an environmental sensor that detects the movement of the operator. The operation terminal 300 is not limited to being provided in a company facility, but may also be a terminal provided in the operator's home, or a terminal provided in a teleworkstation installed in a station, department store, public facility, internet cafe, etc.

[0022] The robot manufacturer terminal 400 is a terminal used, for example, by a manufacturer or management company of the robot device 200. The robot manufacturer terminal 400, for example, updates the version of software (programs) installed in each robot device 200, acquires error information from the robot device 200, and performs remote operation or stop control of the robot device 200 based on the acquired error information.

[0023] The telecommunications company terminal 500 is, for example, a terminal used by a telecommunications company or the like that manages communications on the network NW. The telecommunications company terminal 500 manages the amount of data communicated for remote control of the remote control system 1, manages communication delays, etc., and performs maintenance of the communication environment within the system. The telecommunications company terminal 500 may also take measures when a failure occurs on the network NW.

[0024] Next, the functions of the management server 100, the robot device 200, and the operation terminal 300 will be specifically described. [Administration Server] FIG. 2 is a diagram illustrating an example of the configuration of a management server 100 according to an embodiment. The management server 100 includes, for example, a communication unit 110, a communication control unit 120, an acquisition unit 130, a management unit 140, a control unit 150, and a storage unit 170. The communication control unit 120, the acquisition unit 130, the management unit 140, and the control unit 150 are each implemented by a hardware processor, such as a central processing unit (CPU), executing a program (software). Some or all of these components may be implemented by hardware (including circuitry), such as a large-scale integration (LSI), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU), or may be implemented by a combination of software and hardware. Some or all of the functions of these components may be implemented by a dedicated LSI. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as a hard disk drive (HDD) or flash memory provided in the management server 100, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and the storage medium may be installed in the HDD or flash memory provided in the management server 100 by being attached to a drive device provided in the management server 100. The management server 100 may be realized in a server device or storage device incorporated in a cloud computing system. In this case, the functions of the management server 100 may be realized by multiple server devices and storage devices in the cloud computing system.

[0025] The communication unit 110 communicates with one or more robot devices 200, an operation terminal 300 that remotely controls one or more robot devices 200, and other external devices via the network NW under the control of the communication control unit 120. The communication control unit 120 controls communication in the communication unit 110.

[0026] The communication control unit 120 controls communication in the communication unit 110. For example, the communication control unit 120 receives operation details of the robot device 200 input from the operation terminal 300, and transmits operation control information corresponding to the operation details to the robot device 200. The communication control unit 120 also receives information such as detection results of sensors or camera images transmitted from the robot device 200, and transmits information to be provided based on the received information to the operation terminal.

[0027] The acquisition unit 130 acquires various types of information from the robot device 200 or the operation terminal 300. For example, the acquisition unit 130 acquires operation information for operating the robot device 200 from the operation terminal 300. The acquisition unit 130 also acquires information from sensors and cameras provided in the robot device 200. The acquisition unit 130 may also acquire information from external devices connected to the network NW.

[0028] The management unit 140 includes, for example, an authentication management unit 142 and an information management unit 144. The authentication management unit 142 manages the authentication and usage authority of the operator (system user) who operates the robot device 200. Furthermore, the authentication management unit 142 associates and manages the robot device 200 operated by the operation terminal 300 based on the authentication result.

[0029] The information management unit 144 manages information available to each user. For example, the information management unit 144 stores robot control information for controlling the robot and information (sensor information) acquired from a sensor provided in the robot device 200 in the storage unit 170. The information management unit 144 may set different access rights for each information, and when the set access rights are confirmed, transmit the information to the operation terminal 300 that has the corresponding authority to provide the information. The information management unit 144 may also set stronger access rights for the sensor information than for the robot control information. Strengthening the access rights means, for example, storing the sensor information in a storage area that cannot be accessed unless a specific authentication is granted, encrypting and managing the data itself, or increasing the number and types of authentication conditions, such as passwords, for acquiring the data. This ensures data confidentiality. Therefore, for example, it is possible to prevent theft or tampering of sensor information by companies that do not have access rights, thereby improving the security of the entire system. Furthermore, by managing the information provided based on access rights, various companies and the like can use the remote operation system 1 with peace of mind.

[0030] The control unit 150 includes, for example, a robot information control unit 151, a customized data control unit, an intention estimation unit 153, a movement control unit 154, a movement cancellation unit 155, and an output control unit 156. The robot information control unit 151 manages, for example, what operations the robot device 200 actually performed in response to the movement control information for the robot device 200 generated by the movement control unit 154. The robot information control unit 151 also causes the robot device 200 to perform basic movements based on the robot control information 172A stored in the storage unit 170. The basic movements are, for example, movements to move the robot device 200, change its posture, and pick up or place an object.

[0031] The customization data control unit 152 generates customization data for causing the robot device 200 to perform specific operations (e.g., assembling an object, soldering, screwing, various cooking processes, etc.) for each company.

[0032] The intention estimation unit 153 estimates the operator's intention from at least a portion of the information on the operation content input via the operation terminal 300. The operator's intention includes, for example, not only the operator's intention of the action (movement) that the operator wants the robotic device 200 to perform, but also prediction of the action and assistance in driving the robotic device 200. For example, in remote control, a discrepancy occurs between the sense of the operator's instructions via communication and the actual movement of the robotic device 200. Therefore, the intention estimation unit 153 does not simply cause the robotic device 200 to execute the operation content from the operator, but estimates the operator's intention based on the operation content and generates movement control information for the robotic device 200 based on the estimation result, thereby more appropriately assisting the movement of the robotic device 200 and realizing remote control with less discomfort. The intention estimation unit 153 estimates the intention corresponding to the operation content by, for example, referring to a learning model (intention estimation information 172B) trained using training data. Note that during learning, for example, the operation content is input and training data is used to output correct intention information set by a worker or the like for the operation content.

[0033] The intention estimation unit 153 may also update the intention estimation information 172B based on history information indicating the operation content input by the operator and the actual operation content of the robot device 200. This allows for more appropriate intention estimation.

[0034] The intention estimation unit 153 may also estimate the operator's intention by, for example, the GRASP Taxonomy method (see, for example, Reference 1). In the embodiment, the operator's state is classified by classifying the posture, i.e., the grip posture, of the operator or the robotic device 200 by, for example, the GRASP Taxonomy method, and the operator's intention is estimated. Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77.

[0035] The operation control unit 154 generates operation control information for operating the target robot device 200 based on the operator's intention estimated by the intention estimation unit 153, transmits the generated operation control information to the target robot device 200, and controls the operation of the robot device 200. Furthermore, in addition to (or instead of) the operator's intention, the operation control unit 154 may acquire information on the operation of the robot device 200 obtained by the robot information control unit 151, the customized data control unit 152, etc., and generate the operation control information based on the acquired information. Furthermore, for example, when multiple operators operate two arms simultaneously, the operation control unit 154 may perform synthesis processing such as addition or subtraction of forces based on sensor information such as tactile information, and perform operation control for cooperative work. This makes it possible to achieve well-balanced operations even when one robot device 200 is operated by multiple operators.

[0036] The action cancellation unit 155 selects an action that is different from the operator's intention from among the actions that the action control unit 154 causes the robot device 200 to perform, and cancels the execution of the selected action. In this case, the action cancellation unit 155 inquires of the operator whether or not the operation content input by the operator is the action that the robot device 200 is to perform, and based on the inquiry result, causes the action control unit 154 to generate action control information and update the intention estimation information 172B.

[0037] The output control unit 156 generates information to be provided to the operator operating the operation terminal 300 based on the audio and images acquired from the robot device 200. The information to be provided includes images and audio. The output control unit 156 also outputs sensor information and the like based on the result of the authentication process by the authentication management unit 142.

[0038] The storage unit 170 may be realized by any of the above-mentioned storage devices, or a solid-state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), or a random-access memory (RAM). The storage unit 170 includes, for example, a first storage unit 172 and a second storage unit 174. The first storage unit 172 and the second storage unit 174 are storage areas with different levels of access rights. The first storage unit 172 stores, for example, information available to all users of the remote control system 1. The second storage unit 174 stores, for example, information available only to companies or users whose access rights have been confirmed in advance. The second storage unit 174 may be an area where encrypted data for each company is stored and cannot be viewed without decryption using a key assigned to each company. The access rights set by the first storage unit 172 and the second storage unit 174 are managed by the authentication management unit 142. Furthermore, the first storage unit 172 and the second storage unit 174 may be different storage units, and one or both may be managed by an external device such as a data server.

[0039] For example, the first storage unit 172 stores robot control information 172A and intention estimation information 172B. The robot control information 172A is basic control information for causing the robot device 200 to execute a predetermined action. The predetermined action is, for example, an action such as moving, changing direction, moving an arm, holding (grasping) an object, or putting down an object. These may be set for each type of robot device 200, for example.

[0040] The intention estimation information 172B is, for example, information about the intention estimation result estimated by the intention estimation unit 153. Fig. 3 is a diagram for explaining the contents of the intention estimation information 172B. In the intention estimation information 172B, for example, information about an estimated intention is associated with the operation content acquired from the operation terminal 300. This information may be managed for each operator, or the intention estimation information 172B of multiple operators may be integrated to store only information about frequently occurring intention estimations, or may be updated based on history information, etc.

[0041] Information accessible to a user whose access right has been confirmed is stored in the second storage unit 174. For example, the second storage unit 174 stores company-specific user information 174A, company-specific sensor information 174B, and the like.

[0042] FIG. 4 is a diagram illustrating the contents of the company-specific user information 174A. In the example of FIG. 4, user information for each company is stored, and each piece of information is managed so that it cannot be accessed unless the access right of each company is confirmed. The company-specific user information 174A is, for example, authentication information for authenticating a user when using a service provided by the remote operation system 1, associated with information such as name, authority, priority, and image information. The authentication information includes, for example, a user ID and password, which are identification information for identifying a user. The authentication information may also include biometric authentication information such as fingerprint information and iris information. The authority includes, for example, authority information such as access rights assigned to a user. Sensor information acquired by the robotic device 200 can be acquired based on this authority information. The information management unit 144 can manage who can access which data and which data cannot be accessed by referring to the information stored in the authority. The priority information stores, for example, priorities for when multiple operators remotely operate the robotic device 200 and the operations performed by each operator are the same or similar. Image information is an image for recognizing a user, and includes, for example, a facial image of the user, an avatar image created to resemble the user, an illustrated image, and the like.

[0043] FIG. 5 is a diagram illustrating the contents of the company-specific sensor information 174B. In the example of FIG. 5, sensor information for each company is stored, and each piece of information is managed so that it cannot be accessed unless the access right of each company is confirmed. In the company-specific sensor information 174B, a robot ID, which is identification information for identifying the robot device 200, is stored with a date and time, an operator ID, operation parts, sensor information, etc. The operator ID stores the user ID of the operator who performed the operation. The operation parts store the parts of the robot ID operated by the operator. This makes it possible to manage the operation history of each operator even when one robot device 200 is operated by multiple operators.

[0044] [Robot device] 6 is a diagram illustrating an example of the configuration of a robot device 200 according to an embodiment. The robot device 200 includes, for example, a communication unit 202, a sensor 204, a camera (an example of an imaging unit) 206, a microphone 208, a driving unit 210, a monitor 214, a speaker 216, a control device 240, and a storage unit 260. The driving unit 210 includes, for example, an arm unit 210A and a movement driving unit 210B. The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are each realized by a hardware processor, such as a CPU, executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (storage device having a non-transitory storage medium) such as an HDD or flash memory provided in the robot device 200, or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM, and installed in the HDD or flash memory provided in the robot device 200 by attaching the storage medium to a drive device provided in the robot device 200.

[0045] The communication unit 202 will be described later. Under the control of the communication control unit 242, the communication unit 202 communicates with the management server 100 and other external devices via the network NW. The communication unit 202 may also communicate with other robotic devices 200, or may communicate with the operation terminal 300.

[0046] The sensor 204 includes a position sensor that detects the position of the robotic device 200, a speed sensor that detects the speed, and a temperature sensor that detects the temperature at a specific position such as the periphery of the robotic device 200 or the tip of the arm unit 210A. The position sensor receives information from, for example, a GPS (Global Positioning System) receiving device and determines position information (longitude and latitude information) based on the received information. The sensor 204 may also include a humidity sensor that detects the ambient humidity and a vibration sensor that detects vibrations of an object operated by the robotic device 200. The sensor 204 may also include an object detection sensor that detects surrounding objects. Examples of surrounding objects include other robotic devices 200, people, and obstacles.

[0047] The camera 206 is, for example, a digital camera using a solid-state image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 206 captures an image of the periphery of the robotic device 200 at a predetermined timing, for example. The number of cameras 206 is not limited to one, and multiple cameras may be provided in the robotic device 200. For example, the camera 206 is provided in the head of the robotic device 200. The camera 206 may also be provided near the tip of the arm unit 210A. This allows the object to be worked on to be photographed at a close distance, making it easier to perform more detailed work.

[0048] The microphone 208 receives sounds from around the robot device 200. The microphone 208 outputs information based on the input sounds to the control device 240.

[0049] The arm unit 210A grasps a target object and performs a predetermined task on the target object. The arm unit 210A is, for example, a multi-joint robot arm and includes, for example, an actuator, a gear, an artificial muscle, and the like. For example, the arm unit 210A includes a first arm unit, one end of which is connected to the right side of the robot device body, and a second arm unit, one end of which is connected to the left side of the robot device body. The arm unit 210A may also include other arm units in addition to the first and second arm units. Hereinafter, when there is no need to distinguish between the arms, they will simply be referred to as the "arm unit 210A." The other end of the arm unit is configured with a gripping unit capable of gripping a predetermined object. The arm unit 210A is driven based on the control of the control device 220. The arm unit is capable of movements similar to those of a human arm.

[0050] The movement drive unit 210B is a drive unit for moving the robot device 200 on the floor or ground. The movement drive unit 210B may be, for example, two legs, or may be provided with a movement mechanism such as wheels, a cart, or a crawler. For example, if the movement drive unit 210B is a leg, the leg operates to make the robot device 200 walk under the control of the control device 220. The movement drive unit 210B may also have a structure that can move along a ceiling or rail by driving a crane or the like. With this configuration, the robot device 200 can move in a desired direction. Note that the drive unit 210 may be provided with a mechanism for driving other joints, such as the waist or head, in addition to the arm unit 210A or the movement drive unit 210B. The drive unit 210 is driven under the control of the drive control unit 246.

[0051] The monitor 214 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The monitor 214 displays the information output by the output control unit 248 as an image. A plurality of monitors 214 may be provided in the robot device 200. The monitor 214 may be provided, for example, on the head, abdomen, or back. The speaker 216 outputs the information output by the output control unit 248 as sound.

[0052] The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuit units) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (non-transitory storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device.

[0053] The communication control unit 242 communicates with the management server 100, for example, by wireless communication via the communication unit 202, to send and receive information. The communication control unit 242 may also communicate with other robotic devices 200 via the communication unit 202.

[0054] The acquisition unit 244 acquires the detection results (sensor information) of the sensor 204 provided in the robot device 200. The acquisition unit 244 also acquires images captured by the camera 206. The acquisition unit 244 also acquires information based on sounds input from the microphone 208.

[0055] The drive control unit 246 operates the arm unit 210A and the movement drive unit 210B based on the operation control information acquired from the management server 100. Furthermore, the drive control unit 246 drives other parts of the robot device 200 (e.g., the head, torso, waist, etc.) based on the operation control information to change the posture, etc. Furthermore, the drive control unit 246 may drive the robot device 200 based on basic operation information 262 and customized operation information 264 stored in the storage unit 260. The basic operation information 262 is drive control information for causing the robot device 200 to perform a basic operation according to the type, etc., of the robot device 200. The customized operation information 264 is drive control information for causing the robot device 200 to perform a specific operation that is registered in advance for each operator, for example.

[0056] The output control unit 248 causes the monitor 214 to output an image based on the information acquired from the management server 100. The output control unit 248 also causes the speaker 216 to output a voice, a warning sound, or the like based on the information acquired from the management server 100.

[0057] The storage unit 260 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, a RAM, etc. The storage unit 260 stores, for example, basic operation information 262, customized operation information 264, programs, and other information.

[0058] [Operation terminal] 7 is a diagram showing an example of the configuration of an operation terminal 300 according to the embodiment. The operation terminal 300 includes, for example, a communication unit 310, an HMD 320, an operation device 330, an environmental sensor 340, a control device 360, and a storage unit 380. The HMD 320, the operation device 330, and the environmental sensor 340 are examples of an "HMI (Human Machine Interface)."

[0059] The communication unit 310 communicates with the management server 100 via the network NW under the control of the control device 360. The communication unit 310 may also communicate with other operation terminals 300 and robot devices 200.

[0060] The HMD 320 includes, for example, an image display unit 322, a gaze detection unit 324, a sensor 326, a speaker 328, and a control unit 329. The HMD 320 outputs and displays the status image and sound of the robot device 200 that the control device 360 ​​receives from the management server 100, and detects the movement of the operator's gaze, etc.

[0061] The image display unit 322 is, for example, an LCD, an organic EL display, etc. Under the control of the control unit 329, the image display unit displays an image output by an output control unit 368, which will be described later.

[0062] The gaze detection unit 324 detects the gaze of the operator wearing the HMD 320 under the control of the control unit 329, and outputs the detected gaze information (operator sensor value) to the control device 360. The gaze information is information including, for example, a gaze vector.

[0063] The sensor 326 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 326 detects the tilt and rotation of the head of the operator wearing the HMD 320, and outputs the detected head movement information (operator sensor value) to the control device 360.

[0064] The speaker 328 outputs the voice, warning sound, etc. output by the output control unit 368.

[0065] The control unit 329 controls the execution of gaze detection by the gaze detection unit 324 based on control information from the control device 360, controls detection by the sensor 326, controls image display on the image display unit 322, and accepts information input from the operator via the HMD 320.

[0066] The operation device 330 includes, for example, a sensor (operator sensor) 332, a control unit 334, a feedback unit 336, and an input unit 338. The operation device 330 includes, for example, a tactile data glove that is worn on the hand of the operator.

[0067] The sensor 332 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 332 includes a plurality of sensors. The sensor 332 tracks the movement of each finger using, for example, two sensors. The sensor 332 detects operator arm information (operator sensor value) that is information related to the orientation and movement of each finger of the operator, the posture and position of the operator's arm, such as the movement of the hand, under the control of the control unit 334, for example. The operator arm information (operator sensor value) includes information related to the entire human arm, such as hand position and posture information, angle information of each finger, position and posture information of the elbow, and information tracking the movement of each part.

[0068] The control unit 334 outputs the operator's arm information detected by the sensor 332 to the control device 360. Furthermore, the control unit 334 controls the feedback unit 336 based on feedback information acquired from the control device 360.

[0069] The feedback unit 336 feeds back feedback information to the operator in accordance with the control of the control unit 334. In accordance with the feedback information, the feedback unit 336 feeds back sensations to the operator using, for example, a means for applying vibration (not shown), a means for applying air pressure (not shown), a means for restricting hand movement (not shown), a means for making the operator feel temperature (not shown), a means for making the operator feel hardness or softness (not shown), or a means for making the operator feel vibration (not shown), which are attached to the arm unit 210A of the robot device 200.

[0070] The input unit 338 is, for example, an input device other than the tactile data glove, such as a keyboard, a mouse, a lever, a touch panel, a microphone, etc. The input unit 338 receives input of operation details for the robot device 200 from each input device.

[0071] The environmental sensor 340 detects, for example, the movement of the operator. The environmental sensor 340 includes, for example, a camera (an example of an imaging unit) 342, a sensor 344, and an object position detection unit 346. The camera 342 captures an image including the operator. The camera 342 is, for example, an RGB camera. The camera 342 outputs the captured image to the object position detection unit 346. Note that in the environmental sensor 340, the positional relationship between the camera 342 and the sensor 344 is known.

[0072] The sensor 344 is, for example, a depth sensor. The sensor 344 outputs the detection result to the object position detection unit 346. Note that the camera 342 and the sensor 344 may be distance sensors.

[0073] The object position detection unit 346 detects the three-dimensional position, size, shape, etc. of the target object in the captured image using a known method, based on the image captured by the camera 342 and the detection result detected by the sensor 344. The object position detection unit 346 estimates the position of the object by performing image processing (edge ​​detection, binarization processing, feature extraction, image enhancement processing, image extraction, pattern matching processing, etc.) on the image captured by the camera 342 with reference to a pattern matching model stored in the object position detection unit 346. If multiple objects are detected in the captured image, the object position detection unit 346 detects the position of each object. The object position detection unit 346 transmits the detected object position information to the control device 360. The data transmitted by the environmental sensor 340 may be, for example, a point cloud having position information.

[0074] The control device 360 ​​includes, for example, a communication control unit 362, an acquisition unit 364, an operation content generation unit 366, and an output control unit 368. The communication control unit 362, the acquisition unit 364, the operation content generation unit 366, and the output control unit 368 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuit units) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (non-transitory storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device.

[0075] The communication control unit 362 communicates with the management server 100 by wireless communication via the communication unit 310, for example, to send and receive information. The communication control unit 362 may also communicate with other operation terminals 300 via the communication unit 310. The communication control unit 362 also communicates with the HMD 320, the operation device 330, and the environmental sensor 340.

[0076] The acquisition unit 364 acquires information obtained from the HMD 320, the operation device 330, and the environmental sensor 340. The operation content generation unit 366 generates operation content for the robotic device 200 based on the information acquired by the acquisition unit 364. For example, the operation content generation unit 366 generates operation content related to gaze information and head direction based on information acquired from the HMD 320. The operation content generation unit 366 also generates operation content related to the movement of the arm unit 210A of the robotic device 200 based on information acquired from the operation device 330. The operation content generation unit 366 also generates operation content related to the posture, movement direction, and movement amount of the robotic device 200 based on information acquired from the environmental sensor 340. The information generated by the operation content generation unit 366 is transmitted to the management server 100 via the communication unit 310.

[0077] The output control unit 368 outputs the provided information acquired from the management server 100 to the HMD 320 and the operation device 330. For example, the output control unit 368 causes an image acquired from the management server 100 to be output to the image display unit 322 of the HMD 320. The output control unit 368 also causes a speaker 328 to output a voice, a warning sound, or the like based on the information acquired from the management server 100. The output control unit 368 also causes the operation device 330 to output feedback information for conveying tactile information acquired from the management server 100 to the operator.

[0078] The storage unit 380 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, or a RAM, etc. The storage unit 380 stores, for example, programs and other information.

[0079] [Regarding remote control of the robot device according to the embodiment] Next, remote operation of the robot device 200 by the remote operation system 1 according to the embodiment will be specifically described. FIG. 8 is a diagram for explaining the remote operation of the robot device 200 according to the present embodiment. FIG. 8 shows an example in which one robot device 200 is operated based on the operations of two operators U1 and U2. As shown in FIG. 8, the operators U1 and U2 each wear an HMD 320 and an operation device 330. The operation terminals 300A-1 and 300A-2 are terminals of the same company (for example, company A), but may be installed in different locations.

[0080] In addition, the operators U1 and U2 are equipped with environmental sensors 340 that measure the movements of the operators and the surrounding environment. Note that a configuration similar to the environmental sensor 340 may be attached to, for example, the robotic device 200. The operation device 330 includes, for example, an operation device 330a worn on the left hand of the operator and an operation device 330b worn on the right hand. The operation contents input from each of the operation devices 330a and 330b are used to control the movements of the left hand LH and right hand RH of the robotic device 200. The position of the operator's arms, the direction of the operator's face, etc. are detected by the environmental sensors 340. The detected data is transmitted to the management server 100 via a network.

[0081] The management server 100 estimates the intention of each user based on the operation content obtained from the operation terminals 300A-1 and 300A-2. An example of information estimated by the intention estimation unit 153 will now be described in detail. The intention estimation unit 153 estimates the operator's intention based on the acquired operation content from the operator. The intention estimation unit 153 classifies the posture of the arm including the gripping unit of the robot device by classifying the posture of the operator's arm based on the operator sensor value of the operation terminal 300. Furthermore, the intention estimation unit 153 estimates the operator's intention to have the robot device 200 perform based on the classification result. The intention estimation unit 153 estimates, as the operator's intention, hand and finger movements at each time, the purpose of the task that the robot device 200 is to perform, the task content, and hand and finger movements at each time. The task purpose is, for example, grasping an object, moving an object, etc. The task content is, for example, grasping and lifting an object, grasping and moving an object, etc.

[0082] Furthermore, the intention estimation unit 153 may estimate the operator's intention by inputting the operation details into, for example, a learned model or intention estimation information 172B stored in the storage unit 170. In the embodiment, the intention estimation is performed based on the classification of the grip posture, thereby enabling the operator's intention to be estimated with high accuracy. Note that other methods may be used to classify the grip posture.

[0083] Alternatively, the intention estimation unit 153 may perform an integrated estimation using the gaze and the arm movement. In this case, the intention estimation unit 153 may input gaze information, hand movement information, and position information of the object OB on the table TB into a trained model to estimate the operator's intention.

[0084] In the example of FIG. 8 , the intention estimation unit 153 estimates the object to be grasped based on, for example, gaze information. Next, the intention estimation unit 153 estimates the posture of the operator's hand based on the estimated object to be grasped. Alternatively, the intention estimation unit 153 may first estimate the posture of the operator's hand based on the operation content, and then estimate the object to be grasped from the estimated hand posture of the operator. Note that when the same or similar operation content is input from operators U1 and U2, intention estimation is performed based on the operation content of the operator with the higher priority. Furthermore, when operations of different parts are assigned to operators U1 and U2, the intention estimation unit 153 estimates the intention based on the operation content of each operator. In the following description, it is assumed that operator U1 operates the right-hand arm unit 210A and operator U2 operates the left-hand arm unit 210A.

[0085] For example, when an object OB is placed on a table TB shown in Fig. 8, the intention estimation unit 153 estimates that the object OB will be grasped based on the posture of the hand. The intention estimation unit 153 may also estimate in advance the future trajectory of the hand intended by the operator based on the operation content and state information of the robot device 200. Note that the intention estimation unit 153 may also estimate the object to be grasped and its position by using the detection results detected by the sensors, the results of image processing of images captured by the environmental sensor 340, and the like.

[0086] Furthermore, since the coordinate systems of the environment in which the operator operates and the robot operating environment are different, calibration between the operator operating environment and the robot operating environment may be performed, for example, when starting up the robot device 200. Furthermore, when grasping an object, the management server 100 may determine the grasping position based on the gripping force of the robot device and the frictional force between the object and the gripping unit, taking into account the error in the gripping position at the time of grasping.

[0087] When the robot device 200 is not remotely operated, its operation is controlled according to the control of the control device 240, and when it is remotely operated, its operation is controlled according to operation control information generated by the management server 100 based on the estimated intention, etc.

[0088] The control device 240 of the robot device 200 controls the driving unit 210 based on operation control information from the management server 100. The robot device 200 also outputs information to the management server 100, such as sensor information detected by the sensor 204, images captured by the camera 206, and audio output from the microphone 208. The management server 100 generates information to be provided to the operation terminals 300A-1 and 300A-2 based on the acquired sensor information, images, audio, and the like. In this case, the management server 100 may generate an image in which at least a portion of the sensor information is superimposed. The management server 100 generates information to be provided that also includes information to be transmitted via the sense of touch. Furthermore, since the robot device 200 can be operated by multiple operators, an image may be generated in which information identifying the operator of each part of the robot device 200 is superimposed on a camera image. The operation terminal 300 displays the image output from the management server 100.

[0089] FIG. 9 is a diagram showing an example of an image IM10 displayed by the operation terminal 300. Note that the display mode of the image IM10, such as the layout and display content, is not limited to the example shown in FIG. 9. The same applies to the display modes of other images described below. In the image IM10 shown in FIG. 9, a sensor information display area A11, an operator information display area A12, and a work status display area A13 are superimposed on an image IM10 including the arm portion (left hand LH, right hand RH) of the robot device 200 captured by the camera 206. In the example shown in FIG. 9, an object OB1 is placed on a table TB in front of the robot device 200. The left hand LH of the robot device 200 is holding and lifting an object OB2, and the right hand RH of the robot device 200 is holding and lifting an object OB3. The sensor information display area A11, the operator information display area A12, and the work status display area A13 may be displayed with a predetermined transmittance so that objects in the real space are not blocked.

[0090] In the example of FIG. 9, the sensor information display area A11L displays sensor information related to the left hand of the robotic device 200. Furthermore, the sensor information display area A11R displays sensor information (e.g., position, pressure, temperature, etc.) related to the right hand of the robotic device 200. Furthermore, in the example of FIG. 9, the operator information display area A12L displays information related to the operator of the left hand of the robotic device 200. The operator information display area A12R displays information related to the operator of the left hand of the robotic device 200. This allows, for example, when the robotic device 200 is operated by multiple people, the other operators to accurately know who is operating each part of the robotic device 200. Furthermore, since it is possible to know not only the sensor information of the part that one operator is operating but also information related to the operation of other operators, more accurate processing can be performed.

[0091] The work status display area A13 displays the results of each intention estimation, etc. In the example of FIG. 9, information about the intentions estimated from the operation details of each of the operators U1 and U2 is displayed. Furthermore, when the management server 100 estimates through intention estimation that each operator will perform the same or similar process, it prioritizes the execution of one intention based on priority information. In this case, the work status display area A13 may be displayed to the other operator, and a warning may be presented by changing the audio or display format. In the example of FIG. 9, because both operators U1 and U2 are attempting to place the objects they are holding on object OB1, warning information is presented. The warning format may be varied depending on the degree of danger.

[0092] In this case, the operation control unit 154, for example, refers to the priority of each operator and causes the robot device 200 to perform an operation based on the intention of the operator with the highest priority. Furthermore, when a process is performed based on the priority, the management server 100 causes the display unit of the operation terminal 300 to display an image including the process content.

[0093] FIG. 10 is a diagram showing an example of an image IM20 displayed on the operation terminal 300 when a task based on priority is executed. In the example of FIG. 10, the content displayed in the task status display area A13 is different from that of the image IM10 shown in FIG. 9. Therefore, the following mainly describes the task status display area A13. In the example of FIG. 10, the task status display area A13 displays information indicating that the instruction of the first operator U2 has been executed preferentially based on the priority. This allows the operators U1 and U2 to more accurately grasp the intention estimation results for their respective operation instructions and the actual task status.

[0094] The intention estimation unit 153 may store the estimation result in a storage unit and use it for subsequent intention estimations. When detecting an unknown action, the intention estimation unit 153 may inquire of the operator whether to cause the robot device 200 to execute the action.

[0095] FIG. 11 is a diagram showing an example of an image IM30 for inquiring of the operator about an operation to be performed by the robotic device. The image IM30 includes, for example, an operation inquiry information display area A31 and a switch display area A32. The operation inquiry information display area A31 displays information for inquiring whether or not the robotic device 200 needs to perform an operation input by the operation device 330. For example, if a head-touching operation is detected but the intention of this operation cannot be estimated, the intention estimation unit 153 displays an image showing text information such as "A head-touching operation has been detected. Do you want the robot to perform this operation?" on the operation terminal 300, as shown in the operation inquiry information display area A31 in FIG. 11.

[0096] In the switch display area A32, for example, icons IC31 and IC32 are displayed. The icons IC31 and IC32 are, for example, GUI (Graphical User Interface) switches. When the icon IC31 is selected, the management server 100 generates operation control information for causing the robot device 200 to execute the instructed operation. When the icon IC12 is selected, the HMI control unit 180 does not cause the robot device 200 to execute the instructed operation. Furthermore, the management server 100 may execute processing to prevent the robot device from inputting the same operation even if the same operation is input in the future. This makes it possible to prevent the robot from executing an operation that is unconsciously performed due to the operator's habits, etc., and to prevent unnecessary operations.

[0097] Furthermore, operators around the robotic device 200 may feel uneasy because they do not know who is operating the robotic device 200. Therefore, the management server 100 may display information about the operator of the robotic device 200 on a monitor of the robotic device 200. In this case, if the robotic device 200 is operated by multiple operators, the operator of each part may be displayed.

[0098] FIG. 12 is a diagram showing an example of an image IM40 displayed on the monitor 214 of the robotic device 200. The image IM40 includes, for example, one or both of operator information display areas A41 and A42. The operator information display area A41 displays text information about the operator operating the robotic device 200. The operator information display area A41 may also include information about the parts of the robotic device 200 operated by the operators. In the example of FIG. 12, the operator information display area A41 displays text information such as "This robot is operated by operator U2 with his right hand, and by operator U1 with his left hand." The operator information display area A42 displays an avatar image registered in the user DB. This allows nearby operators to more accurately identify the person remotely operating the robot and the parts they are responsible for.

[0099] [Processing Sequence] Next, the processing executed by the remote control system in the embodiment will be described using a sequence diagram. Fig. 13 is a sequence diagram for explaining the processing executed by the remote control system 1 in the embodiment. In the example of Fig. 13, the explanation will be given using the management server 100, the robot device 200, and the operation terminals 300-1 and 300-2. Also, in Fig. 13, an example will be explained in which the robot device 200 is operated by the two operation terminals 300-1 and 300-2.

[0100] The operation terminal 300-1 and the operation terminal 300-2 each access the management server 100 to remotely operate the robot device 200 (steps S100, S102). The management server 100 performs authentication processing to determine whether or not the remote operation system 1 is available based on authentication information and the like input from the operation terminals 300-1 and 300-2 (step S104). The following describes a case where use is permitted through authentication. If use is permitted, the management unit of the management server 100 assigns the robot device 200 to be operated by the operation terminals 300-1 and 300-2 (step S106) and requests activation control of the assigned robot device (step S108).

[0101] The robot device 200 starts up the device in response to a start-up request from the management server 100 (step S110), and outputs the detection results of the sensor of the robot device 200, images captured by the camera, and the like to the management server 100 (step S112).

[0102] The management server 100 acquires the information transmitted from the robot device 200, and generates information (provided information) to be provided to the operation terminals 300-1 and 300-2 based on the acquired information (step S114).The management server 100 then transmits the generated provided information to the operation terminals 300-1 and 300-2, respectively (steps S116 and S118).

[0103] Each of the operation terminals 300-1 and 300-2 receives the information to be provided transmitted from the management server 100, and provides the information by outputting the received information from the HMD, the operation device, etc. (steps S120 and S122).

[0104] Next, the operation terminals 300-1 and 300-2 transmit operation details related to the robot device 200 to the management server 100 (steps S124 and S126). The management server 100 performs intention estimation based on the operation details obtained from each of the operation terminals 300-1 and 300-2 (step S128), generates operation control information based on the intention estimation results (step S130), and transmits the generated operation control information to the robot device 200 (step S132).

[0105] The robot device 200 performs an operation based on the operation control information (step S134), and transmits information obtained from sensors and the like (sensor information), camera images, and the like during or as a result of the operation to the management server 100 (step S136).

[0106] The management server 100 stores the received information in the storage unit 170 (step S138), generates information to be provided for transmitting the acquired information to the operation terminals 300-1 and 300-2 (step S140), and transmits the generated information to be provided to the operation terminals 300-1 and 300-2 (steps S142 and S144). Each of the operation terminals 300-1 and 300-2 outputs the acquired information to provide it to the operator (steps S146 and S148). Thereafter, the processes of steps S124 to S148 are continuously performed until the operation of the robot device 200 is completed.

[0107] <Modification> In the embodiments, part or all of the processing of the management server 100, the robot device 200, and the operation terminal 300 may be realized by AI (Artificial Intelligence) technology. In addition, part or all of the functional units or information stored in the functional units or storage units included in the management server 100, the robot device 200, and the operation terminal 300 may be included in other devices.

[0108] Although the above embodiment has been described with reference to a case where multiple operators operate one robot device 200, a single operator may operate multiple robot devices 200. In this case, sensor information detected by each of the multiple robot devices 200 is displayed on the image display unit 322 of the HMD 320. The management server 100 may also transmit operation history information, intention estimation information, and the like to the robot manufacturer terminal 400. This information can be used for developing the robot manufacturer terminal 400 and products, updating programs, and the like. In this case, a discount or reduction service for system usage may be provided to a company that performs eco-driving of the robot device 200 (e.g., low-power driving, low-load driving, low-communication-volume driving), or to a company that provides a robot control method suited to the operator or telework characteristics. This can encourage users to provide information, which can be useful for improving services.

[0109] In the above embodiment, when a single robotic device 200 is operated by multiple operators, the operation intentions and work status of each operator are displayed. However, for example, the above-described intention estimation results, work status, and information about the operators may be provided between multiple robotic devices 200 that exist within a predetermined distance, or between robotic devices performing collaborative work. By using the intention estimation function between robots, it becomes possible to share awareness more effectively than between humans. Furthermore, since the status of surrounding robotic devices 200 can be acquired, group work, etc., can be performed smoothly.

[0110] The embodiment described above can operate the robot more appropriately by, for example, providing a server with a communication unit that communicates with one or more robotic devices 200 and one or more operation terminals 300 that remotely operate at least one of the one or more robotic devices 200 via a network NW, an acquisition unit that acquires operation content input by the operation terminal 300, an intention estimation unit that estimates the intention of the operator of the operation terminal 300 based on the operation content acquired by the acquisition unit, and an operation control unit that controls the operation of the robotic device 200 based on the intention result estimated by the intention estimation unit.

[0111] Furthermore, according to the above-described embodiment, when teleworking by remotely controlling a robot, various devices can be used more generally and can be operated by a team. Furthermore, while conventional remote control only allowed limited tasks using a limited number of dedicated robots, this embodiment makes it possible for anyone to easily perform teleworking using a highly versatile robot.

[0112] Furthermore, according to the embodiment, robot information and sensor information (information that may include corporate confidentiality) can be managed separately, providing a server capable of handling confidential information. Remote operators can access the same server and share control rights and sensor information for the same robot or different robots. For example, when operating the same robot device, multiple operators can operate different arms, allowing a senior and a junior operator to simultaneously operate the same arm of the same robot and share sensor information such as tactile information to add or subtract forces. Furthermore, when operating different robots, telepathy is possible by sharing the intention estimation function and disclosing each other's intentions. Furthermore, by using the intention estimation function, a warning can be issued to prevent two robots from moving in the same direction or colliding when trying to reach the same object. Furthermore, according to the embodiment, one operator can operate two robots. In this case, for example, if two robots are attempting to carry a large desk, one robot can be brought to the edge of the desk in advance and synchronized with the other robot. Furthermore, if the other robot is brought to the other end and attempts to lift it, the two robots will lift it together through synchronization. Furthermore, in the embodiment, more accurate work can be achieved than when humans call out to each other to perform the work. Also, in the embodiment, when one operator has many robots perform the same work, the supported teleoperation function can absorb the individual differences between the robots and the individual differences between the work objects.

[0113] The above-described embodiment can be expressed as follows. A server computer that communicates with a mobile object and a mobile terminal of a user of the mobile object comprises: a storage device that stores a program; a hardware processor; The hardware processor executes the program stored in the storage device, communicating with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices via a network; Acquire operation details input via the operation terminal; Inferring the intention of the operator of the operating terminal based on the acquired operation content, and updating the intention estimation information; generating motion control information for operating the target robot device based on the estimated intention result and the updated intention estimation information, and controlling the motion of the robot device based on the generated motion control information; the intention includes an intention of an action that the operator wants the robot device to perform and a prediction of the action; the intention estimation information is a learning model trained using training data for inputting operation details for the robot device and outputting correct intention information for the operation details, updating the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robot device; The server is configured as follows:

[0114] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0115] 1... remote operation system, 100... management server, 110, 202, 310... communication unit, 120... communication control unit, 130... acquisition unit, 140... management unit, 150... control unit, 170... memory unit, 200... robot device, 204... sensor, 206... camera, 208... microphone, 210... drive unit, 214... monitor, 216... speaker, 240... control device, 260... memory unit, 300... operation terminal, 320... HMD, 330... operation device, 340... environment sensor, 360... control device, 380... memory unit

Claims

1. a communication unit that communicates with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices via a network; an acquisition unit that acquires operation details input by the operation terminal; an intention estimation unit that estimates an intention of the operator of the operating terminal based on the operation content acquired by the acquisition unit and updates intention estimation information; an operation control unit that generates operation control information for operating the target robot device based on the intention result estimated by the intention estimation unit and the updated intention estimation information, and controls the operation of the robot device based on the generated operation control information; the intention includes an intention of an action that the operator wants the robot device to perform and a prediction of the action; the intention estimation information is a learning model trained using training data for inputting operation details for the robot device and outputting correct intention information for the operation details, the intention estimation unit updates the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robot device. server.

2. the acquisition unit acquires robot control information relating to an operation of the robot device and sensor information detected by a sensor provided in the robot device; a management unit that sets different access rights to the robot control information and the sensor information acquired by the acquisition unit, and enables acquisition of either the robot control information or the sensor information when the set access rights are confirmed; The server of claim 1 .

3. the operation control unit generates the operation control information based on priorities of operators who operate the operation terminals when controlling the operation of one robot device based on operation contents acquired from a plurality of operation terminals.

3. The server according to claim 1 or 2.

4. the intention estimation unit estimates the intention of each of the operators of a plurality of operation terminals that operate the single robot device, an output control unit that outputs an intention estimation result of each of the operators estimated by the intention estimation unit to each of the plurality of operation terminals; The server of claim 3.

5. A server according to any one of claims 1 to 4; the one or more robotic devices; the one or more operation terminals; A remote control system comprising:

6. The computer communicating with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices via a network; Acquire operation details input by the operation terminal; Inferring the intention of the operator of the operating terminal based on the acquired operation content, and updating the intention estimation information; generating motion control information for operating the target robot device based on the estimated intention result and the updated intention estimation information, and controlling the motion of the robot device based on the generated motion control information; the intention includes an intention of an action that the operator wants the robot device to perform and a prediction of the action; the intention estimation information is a learning model trained using training data for inputting operation details for the robot device and outputting correct intention information for the operation details, updating the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robot device; Remote control method.

7. On the computer, communicating with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices via a network; Acquire operation details input by the operation terminal; Inferring the intention of the operator of the operating terminal based on the acquired operation content, and updating the intention estimation information; generating motion control information for operating the target robot device based on the estimated intention result and the updated intention estimation information, and controlling the motion of the robot device based on the generated motion control information; the intention includes an intention of an action that the operator wants the robot device to perform and a prediction of the action; the intention estimation information is a learning model trained using training data for inputting operation details for the robot device and outputting correct intention information for the operation details, updating the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robot device; program.

Images