INFORMATION PROVIDING SERVER, INFORMATION PROVIDING SYSTEM, INFORMATION PROVIDING METHOD, AND PROGRAM

Abstract

To provide information so that a user feels as if the user is in a workspace even in remote operation.SOLUTION: A server includes: a communication unit which communicates with one or more operation terminals over a network; an acquisition unit which acquires information on behavior of an operator of the operation terminal; a management unit which manages a location where the operator is arranged in a space; and an output control unit which arranges information based on information representing the operator and information on the behavior, in a location of the operator in the space managed by the management unit, and outputs the arranged information to the operation terminal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information providing server, an information providing system, an information providing method, and a program. [Background technology]

[0002] Conventionally, there is known a technique for remotely controlling a robot to operate experimental equipment or devices using data transmitted from a monitor terminal or another terminal (see, for example, Patent Document 1). Conventionally, there is also known a technique for displaying real-life images of classmates in an array in each person's evaluation area, or displaying the seat in the line of sight (see, for example, Patent Documents 2 and 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-092749 A [Patent Document 2] Japanese Patent Application Publication No. 10-171337 [Patent Document 3] Patent No. 6007377 Summary of the Invention [Problem to be solved by the invention]

[0004] However, remote operation can cause a sense of loneliness because it is done by one person. Also, no consideration has been given to making the user feel as if another person is actually there during remote operation.

[0005] The aspects of the present invention have been made taking such circumstances into consideration, and one of the objects is to provide an information providing server, an information providing system, an information providing method, and a program that can provide information that makes a user feel as if they are in the work space, even when operating remotely. [Means for solving the problem]

[0006] An information providing server, an information providing system, an information providing method, and a program according to the present invention employ the following configuration. (1): An information providing server according to one embodiment of the present invention is an information providing server including a communication unit that communicates with one or more operation terminals via a network, an acquisition unit that acquires information regarding the behavior of an operator of the operation terminal, a management unit that manages the location of the operator in space, and an output control unit that arranges information based on information representing the operator and information regarding the behavior at the location of the operator in space managed by the management unit, and outputs the arranged information to the operation terminal.

[0007] (2): In the above aspect (1), the management unit generates an image in which the operator selects a placement location, transmits the generated image to the operation terminal, and manages the operator's placement location based on the placement location selected using the image.

[0008] (3): In the above aspect (1) or (2), the operation terminal includes a terminal for remotely operating one or more robot devices.

[0009] (4): In any one of the above aspects (1) to (3), the management unit acquires a position in real space of the robot device operated by the operator, and when outputting an image including the robot device placed at the acquired position to the operation terminal, superimposes an image representing the operator on a display area of ​​the robot device.

[0010] (5): In the above aspects (1) to (3), the management unit manages the location of the operator by associating it with seat information previously assigned in the virtual space, and when an image showing the virtual space is output to the operation terminal, an image representing the operator is superimposed in association with the position of the image of the seat in the virtual space.

[0011] (6): In the aspect described in any one of (1) to (5) above, the output control unit acquires the voice of the operator and outputs the acquired voice by localizing a sound image at the placement location.

[0012] (7): Another aspect of the present invention is an information provision system comprising an information provision server described in any one of (1) to (6) above, the one or more operation terminals, and one or more robot devices remotely operated by the one or more operation terminals.

[0013] (8): Another aspect of the present invention is a method of providing information in which a computer communicates with one or more operation terminals via a network, acquires information regarding the behavior of an operator of the operation terminal, manages a location of the operator in space, places information based on information representing the operator and information regarding the behavior at the managed location of the operator in space, and outputs the placed information to the operation terminal.

[0014] (9): Another aspect of the present invention is a program that causes a computer to communicate with one or more operation terminals via a network, obtain information regarding the behavior of an operator of the operation terminal, manage the location of the operator in space, place information based on information representing the operator and information regarding the behavior at the managed location of the operator in space, and output the placed information to the operation terminal. Effect of the Invention

[0015] According to the above aspects (1) to (9), it is possible to provide information that makes the user feel as if he or she is in the work space even during remote operation.

[0016] According to the above aspect (2), it is possible to place information representing the operator and information based on the information relating to the action at a position in space desired by the operator.

[0017] According to the above aspects (3) and (4), an image representing the operator is displayed at the position where the robot device 200 is displayed, so that it is possible to prevent the operator from feeling lonely.

[0018] According to the above aspect (5), even if the work location is in a virtual space, it is possible to grasp the surrounding situation, and information can be provided that makes the user feel as if they are in the work space.

[0019] According to the above aspect (6), the operator can hear a sound that corresponds to the placement location, making the operator feel as if he or she is in a more realistic working space. [Brief description of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information providing system 1 according to an embodiment. [Diagram 2] 1 is a diagram illustrating an example of a configuration of an information providing server 100 according to an embodiment. [Diagram 3] FIG. 13 is a diagram for explaining the contents of user information 171. [Figure 4] FIG. 13 is a diagram for explaining the contents of utilization status information 172. [Diagram 5] FIG. 13 is a diagram for explaining the contents of intention estimation information 174. [Figure 6] FIG. 13 is a diagram for explaining the contents of sensor information 175. [Figure 7] FIG. 2 is a diagram illustrating an example of a configuration of a robot device 200 according to an embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a configuration of an operation terminal 300 according to the embodiment. [Figure 9] FIG. 2 is a diagram for explaining remote control of the robot device 200 according to the embodiment. [Figure 10] FIG. 13 is a diagram showing an example of an image IM10 for allowing a user to select a seat. [Figure 11] 1 is a diagram showing an example of an image IM20 captured by the robot device 200. FIG. [Figure 12]FIG. 13 is a diagram for explaining superimposing image information of a user on another robot device 200 (U4) existing in the vicinity. [Figure 13] FIG. 13 is a diagram showing an example of an image IM40 from which images of a worker and a robot device 200 present in the vicinity have been deleted. [Figure 14] 11 is a diagram showing an example in which information representing an operator and information based on information regarding an action are arranged in a virtual space. FIG. [Figure 15] 4 is a sequence diagram for explaining a process executed by the information providing system 1 in the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, an information providing server, an information providing system, an information providing method, and a program according to an embodiment of the present invention will be described with reference to the drawings.

[0022] [System Configuration] FIG. 1 is a diagram showing an example of the configuration of an information providing system 1 according to an embodiment. The information providing system 1 shown in FIG. 1 includes, for example, an information providing server 100, a robot device 200, and an operation terminal 300. The information providing server 100 may include a robot manufacturer terminal 400 and a telecommunications company terminal 500 in addition to the above configuration. The information providing server 100 is communicably connected to the robot device 200, the operation terminal 300, the robot manufacturer terminal 400, and the telecommunications company terminal 500 via a network NW. The network NW includes, for example, the Internet, a cellular network, a Wi-Fi (registered trademark) network, a WAN (Wide Area Network), a LAN (Local Area Network), a provider device, a wireless base station, and the like. The information providing system 1 may include one or more robot devices 200 and one or more operation terminals 300. In the example of FIG. 1, robot devices 200A, 200B, and 200C are shown, but the number and types of the robot devices are not limited to this. 1 shows a plurality of operation terminals 300-1 to 300-n (n is a natural number equal to or greater than 2). In the following description, unless there is a need to distinguish between the robot device and the operation terminal, they will simply be referred to as the "robot device 200" and the "operation terminal 300." The information providing server 100 is an example of a "server."

[0023] The information providing server 100 manages one or more operation terminals 300 and one or more robotic devices 200 to be operated in association with each other. In this case, the information providing server 100 may associate one robotic device 200 with each operation terminal 300, may associate one robotic device 200 with multiple operation terminals 300, or may associate multiple robotic devices 200 with one operation terminal 300. The information providing server 100 acquires operation contents input from the operation terminal 300 and transmits motion control information corresponding to the operation contents to the robotic device 200, thereby remotely operating the robotic device 200. The information providing server 100 transmits information acquired by the robotic device 200 to the operation terminal 300. The information providing server 100 generates information for making the user feel as if he or she is in a working space (for example, around the robotic device 200 to be remotely operated) even during remote operation, and provides the generated information to the operation terminal 300.

[0024] The robot device 200 executes a predetermined operation based on control information transmitted from the information providing server 100 via the network NW. The robot device 200 includes a moving mechanism that can move by driving wheels, a dolly, a crawler, a crane, or the like. In the example of FIG. 1, the robot device 200A is a robot with both arms that can move by at least a crane. The robot device 200B is a robot with both arms that can move by wheels provided at the bottom. The robot device 200C is a robot with both arms that can move by bipedal walking. The robot device 200 also includes an arm unit that performs tasks such as grasping, moving, and operating an object. The robot device 200 also includes a plurality of various sensors such as vibration, temperature, pressure, and tactile sensors, and transmits data detected by each sensor to the information providing server 100 at a predetermined cycle or at the timing when a request is received. The robot device 200 may also include a camera that captures the surroundings, a monitor that displays images, a speaker that outputs sound, a microphone that captures surrounding sounds, and the like.

[0025] The operation terminal 300 inputs the operation contents of the robot device 200 and notifies the operator of information acquired by the robot device 200. The operation terminal 300 can operate one or more robot devices 200. The operation terminal 300 may also operate some parts of the robot device 200 (for example, the right arm or the left arm, etc.). The operation terminal 300 may also be provided with, for example, a visual device such as an HMD (head mounted display), an operation device, and an environmental sensor that detects the movement of the operator. Note that the operation terminal 300 is not limited to being provided in a company, for example, and may be a terminal provided in the operator's home, or a terminal provided in a telework station provided in a station, department store, public facility, internet cafe, etc.

[0026] The robot manufacturer terminal 400 is, for example, a terminal used by a manufacturer or a management company of the robot device 200. The robot manufacturer terminal 400, for example, updates the version of software (programs) installed in each robot device 200, acquires error information from the robot device 200, and performs remote operation or stop control of the robot device 200 based on the acquired error information.

[0027] The communication company terminal 500 is, for example, a terminal used by a communication carrier or the like that manages communication on the network NW. The communication company terminal 500 manages the amount of data communicated for remote operation of the information providing system 1, manages communication delays, and performs maintenance of the communication environment within the system. The communication company terminal 500 may also handle cases where a failure occurs on the network NW.

[0028] Next, the functions of the information providing server 100, the robot device 200, and the operation terminal 300 will be specifically described. [Information server] FIG. 2 is a diagram showing an example of the configuration of the information providing server 100 according to the embodiment. The information providing server 100 includes, for example, a communication unit 110, a communication control unit 120, an acquisition unit 130, a management unit 140, a control unit 150, and a storage unit 170. The communication control unit 120, the acquisition unit 130, the management unit 140, and the control unit 150 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (storage device equipped with a non-transient storage medium) such as a hard disk drive (HDD) or flash memory equipped in the information providing server 100, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and the storage medium may be installed in the HDD or flash memory equipped in the information providing server 100 by being attached to a drive device equipped in the information providing server 100. The information providing server 100 may be realized in a server device or a storage device incorporated in a cloud computing system. In this case, the functions of the information providing server 100 may be realized by a plurality of server devices and storage devices in the cloud computing system.

[0029] The communication unit 110 communicates with one or more robot devices 200, an operation terminal 300 that remotely operates one or more robot devices 200, and other external devices via a network NW under the control of the communication control unit 120. The communication control unit 120 controls the communication in the communication unit 110.

[0030] The communication control unit 120 controls the communication in the communication unit 110. For example, the communication control unit 120 receives the operation content of the robot device 200 input from the operation terminal 300, and transmits operation control information corresponding to the operation content to the robot device 200. The communication control unit 120 also receives information such as the detection results of sensors and camera images transmitted from the robot device 200, and transmits to the operation terminal information to be provided based on the received information.

[0031] The acquisition unit 130 acquires various information from the robot device 200 or the operation terminal 300. For example, the acquisition unit 130 acquires operation information for operating the robot device 200 from the operation terminal 300. The acquisition unit 130 also acquires information on the behavior of the operator from the operation terminal 300. The information on the behavior includes, for example, information on the motion of the operator and information on the voice of the operator. The acquisition unit 130 also acquires information from a sensor or a camera provided in the robot device 200. The acquisition unit 130 may also acquire information from an external device connected to the network NW.

[0032] The management unit 140 includes, for example, an authentication management unit 142, an information management unit 144, and a usage status management unit 146. The authentication management unit 142 manages the authentication and usage authority of the operator (system user) who operates the robot device 200. Furthermore, the authentication management unit 142 manages the robot device 200 operated by the operation terminal 300 in association with the operator (system user) based on the authentication result.

[0033] The information management unit 144 manages information available for each user. For example, the information management unit 144 stores robot control information for controlling the robot and information (sensor information) acquired from a sensor provided in the robot device 200 in the storage unit 170, sets different access rights for each, and transmits information to the operation terminal 300 that has the corresponding authority to provide information when the set access rights are confirmed. This protects the confidentiality of data. Therefore, for example, it is possible to prevent information from being stolen or tampered with by a person who does not have the access rights, and the security of the entire system can be improved. In addition, by managing the information to be provided according to the access rights, various users can use the information providing system 1 with peace of mind.

[0034] The usage status management unit 146 manages the usage status of the information providing system 1 for each user. For example, the usage status management unit 146 manages the spatial location for each user. The spatial location is, for example, a work space, and may be a real space or a virtual space. The following mainly focuses on the real space. The usage status management unit 146 also manages the location of the operator relative to a seat previously assigned in the space, and manages the usage status of information provided to the operation terminal 300, etc.

[0035] The control unit 150 includes, for example, a robot information control unit 151, a customized data control unit, an intention estimation unit 153, an operation control unit 154, an operation cancellation unit 155, and an output control unit 156. The robot information control unit 151 manages, for example, what operation the robot device 200 actually performed in response to the operation control information for the robot device 200 generated by the operation control unit 154. The robot information control unit 151 also causes the robot device 200 to perform basic operations based on the robot control information 173 stored in the storage unit 170. The basic operations are, for example, operations to move the robot device 200, change its posture, and hold or place an object. Control information for these basic operations is stored in the robot control information 173.

[0036] The customization data control unit 152 generates customization data for causing the robot device 200 to execute a specific operation (for example, assembling an object, soldering, screwing, various steps in cooking, etc.).

[0037] The intention estimation unit 153 estimates the intention of the operator from at least a part of the information of the operation contents inputted by the operation terminal 300. The intention of the operator includes, for example, not only the intention of the action (motion) that the operator wants the robot device 200 to perform, but also prediction of the action and assisting the driving of the robot device 200. For example, in remote operation, a discrepancy occurs between the feeling that the operator is instructing through communication and the like and the motion of the robot device 200. Therefore, the intention estimation unit 153 does not make the robot device 200 directly execute the operation contents from the operator, but estimates the intention of the operator based on the operation contents and generates motion control information for the robot device 200 based on the estimation result, thereby more appropriately assisting the movement of the robot device 200 and realizing remote operation with less discomfort. For example, the intention estimation unit 153 refers to a learning model (intention estimation information 174) that is trained using teacher data, and estimates the intention corresponding to the operation contents. Note that, during learning, for example, the operation contents are inputted, and learning is performed using teacher data for outputting correct intention information set by the operator or the like for the operation contents.

[0038] Furthermore, the intention estimation unit 153 may update the intention estimation information 174 based on history information indicating the operation content input by the operator and the actual operation content of the robot device 200. This makes it possible to realize more appropriate intention estimation.

[0039] The intention estimation unit 153 may estimate the intention of the operator, for example, by using the Grasp Taxonomy method (for example, see Reference 1). In the embodiment, the intention of the operator is estimated by classifying the operator state by classifying the posture, i.e., the gripping posture, of the operator or the robotic device 200, for example, by using the Grasp Taxonomy method. Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77.

[0040] The operation control unit 154 generates operation control information for operating the target robot device 200 based on the intention of the operator estimated by the intention estimation unit 153, transmits the generated operation control information to the target robot device 200, and controls the operation of the robot device 200. The operation control unit 154 may acquire information on the operation of the robot device 200 obtained by the robot information control unit 151, the customized data control unit 152, etc. in addition to (or instead of) the intention of the operator, and generate the operation control information based on the acquired information. For example, when multiple operators operate two arm units at the same time, the operation control unit 154 may perform synthesis processing such as addition or subtraction of forces based on sensor information such as tactile sense, and perform operation control for working in cooperation. This makes it possible to realize well-balanced operations even when one robot device 200 is operated by multiple operators.

[0041] The action cancellation unit 155 selects an action different from the operator's intention among the actions to be executed by the robotic device 200 by the action control unit 154, and cancels the execution of the selected action. In this case, the action cancellation unit 155 inquires of the operator whether or not the operation content input by the operator is the action to be executed by the robotic device 200, and causes the action control unit 154 to generate action control information and update the intention estimation information 174 based on the inquiry result.

[0042] The output control unit 156 generates information to be provided to the operator who operates the operation terminal 300, based on the voice and image acquired from the robot device 200. The information to be provided includes images and voice. The output control unit 156 also outputs sensor information and the like, based on the result of the authentication process by the authentication management unit 142. The output control unit 156 also generates information to be provided for each user, based on the contents managed by the usage status management unit 146.

[0043] The storage unit 170 may be realized by the above-mentioned various storage devices, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), a random access memory (RAM), etc. The storage unit 170 stores, for example, user information 171, usage status information 172, robot control information 173, intention estimation information 174, sensor information 175, programs, and other information.

[0044] FIG. 3 is a diagram for explaining the contents of the user information 171. The user information 171 is, for example, authentication information for authenticating a user when using a service provided by the information providing system 1, and information such as a name, authority, priority, and image information associated with the authentication information. The authentication information includes, for example, a user ID and a password that are identification information for identifying a user. The authentication information may also include biometric authentication information such as fingerprint information and iris information. The authority includes, for example, authority information such as an access right assigned to the user. Based on the authority information, the sensor information acquired by the robot device 200 can be acquired. The information management unit 144 can manage who can access which data and which data cannot be accessed by referring to the information stored in the authority. The priority information stores, for example, the priority when a plurality of operators remotely operate the robot device 200 and the operation contents of each are the same or similar. The image information is image information representing a user. The image information includes, for example, a face image or a whole image of the user, an avatar image or an illustration image created to resemble the user so that the user can be identified, and the like. The image information may also include images of the user taken from 360 degrees, or images of only individual parts (for example, arms and legs).

[0045] FIG. 4 is a diagram for explaining the contents of the usage status information 172. In the usage status information 172, for example, a user ID is associated with date and time, seat information, a robot ID which is identification information for identifying the robot device 200, and part information. The date and time may be, for example, a usage start date and time, and may include information such as usage time. The seat information is information on a seat used by a user among the seats arranged in a space where the robot device 200 performs work. This seat may store information on a location (seat) where an actual worker other than the robot device 200 performs work. The part information is information on a part in charge when the robot device is operated by multiple users (operators). The usage status information 172 is managed by the usage status management unit 146 and updated according to the situation.

[0046] 5 is a diagram for explaining the contents of the intention estimation information 174. In the intention estimation information 174, for example, information on an estimated intention is associated with the operation content acquired from the operation terminal 300. These pieces of information may be managed for each operator, or the intention estimation information 174 of a plurality of operators may be integrated to store only information on frequently occurring intention estimation, or may be updated based on history information or the like.

[0047] 6 is a diagram for explaining the contents of the sensor information 175. In the sensor information 175, a robot ID, which is identification information for identifying the robot device 200, is stored with a date and time, an operator ID, operation parts, sensor information, etc. The operator ID is stored with the user ID of the operator who performed the operation. The operation parts are stored with the robot ID parts operated by the operator. This makes it possible to manage the operation history (sensor information) of each operator even when one robot device 200 is operated by multiple operators.

[0048] [Robot device] 7 is a diagram showing an example of the configuration of a robot device 200 according to an embodiment. The robot device 200 includes, for example, a communication unit 202, a sensor 204, a camera (an example of an imaging unit) 206, a microphone 208, a driving unit 210, a monitor 214, a speaker 216, a control device 240, and a storage unit 260. The driving unit 210 includes, for example, an arm unit 210A and a moving driving unit 210B. The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are each realized by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (storage device having a non-transient storage medium) such as an HDD or flash memory provided in the robot device 200, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and installed in the HDD or flash memory provided in the robot device 200 by mounting the storage medium in a drive device provided in the robot device 200.

[0049] The communication unit 202 will be described later. Under the control of the communication control unit 242, the communication unit 202 communicates with the information providing server 100 and other external devices via the network NW. The communication unit 202 may also communicate with other robot devices 200, or may communicate with the operation terminal 300.

[0050] The sensor 204 includes a position sensor that detects the position of the robot device 200, a speed sensor that detects the speed, a temperature sensor that detects the temperature at a specific position such as the periphery of the robot device 200 or the tip of the arm unit 210A, and the like. The position sensor receives information from, for example, a GPS (Global Positioning System) receiving device, and obtains position information (longitude and latitude information) based on the received information. The sensor 204 may also include a humidity sensor that detects the surrounding humidity, and a vibration sensor that detects vibrations of an object operated by the robot device 200. The sensor 204 may also include an object detection sensor that detects surrounding objects. The surrounding objects are, for example, other robot devices 200, people, obstacles, and the like.

[0051] The camera 206 is, for example, a digital camera using a solid-state imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 206 captures an image of the periphery of the robotic device 200 at a predetermined timing, for example. The number of cameras 206 is not limited to one, and a plurality of cameras may be provided in the robotic device 200. For example, the camera 206 is provided in the head of the robotic device 200. The camera 206 may also be provided near the tip of the arm unit 210A. This allows an image of the object to be worked on to be captured at a close distance, making it easier to perform more detailed work.

[0052] The microphone 208 receives sounds from around the robot device 200. The microphone 208 outputs information based on the input sounds to the control device 240.

[0053] The arm unit 210A grasps a target object and performs a predetermined task on the target object. The arm unit 210A is, for example, a multi-joint robot arm, and includes, for example, an actuator, a gear, an artificial muscle, and the like. For example, the arm unit 210A includes a first arm unit in which one end of the multi-joint robot arm is connected near the right side of the robot device main body, and a second arm unit in which one end of the other multi-joint robot arm is connected near the left side of the robot device main body. The arm unit 210A may also include other arm units in addition to the first arm unit and the second arm unit. Hereinafter, when there is no need to distinguish between the respective arms, they will simply be referred to as the "arm unit 210A". The other end of the arm unit is configured with a gripping unit capable of gripping a predetermined object. The arm unit 210A is driven based on the control of the control device 220. The arm unit is capable of movement equivalent to that of a human arm.

[0054] The moving drive unit 210B is a drive unit for moving the robot device 200 on the floor or the ground. The moving drive unit 210B may be, for example, two legs, or may be provided with a moving mechanism such as wheels, a dolly, or a crawler. For example, when the moving drive unit 210B is a leg, the leg operates to make the robot device 200 walk based on the control of the control device 220. The moving drive unit 210B may also be structured to be movable along a ceiling or a rail by driving a crane or the like. With these configurations, the robot device 200 can move in a desired direction. Note that the drive unit 210 may be provided with a mechanism for driving other joints such as a waist or a head in addition to the arm unit 210A or the moving drive unit 210B. The drive unit 210 is driven based on the control of the drive control unit 246.

[0055] The monitor 214 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The monitor 214 displays the information output by the output control unit 248 as an image. A plurality of monitors 214 may be provided in the robot device 200. The monitor 214 may be provided, for example, on the head, abdomen, or back. The speaker 216 outputs the information output by the output control unit 248 as sound.

[0056] The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are realized by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuit units) such as an LSI, ASIC, FPGA, or GPU, or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and may be installed by mounting the storage medium in a drive device.

[0057] The communication control unit 242 communicates with the information providing server 100, for example, by wireless communication via the communication unit 202, to transmit and receive information. The communication control unit 242 may also communicate with other robotic devices 200 via the communication unit 202.

[0058] The acquisition unit 244 acquires the detection result (sensor information) of the sensor 204 provided in the robot device 200. The acquisition unit 244 also acquires an image captured by the camera 206. The acquisition unit 244 also acquires information based on a sound input from the microphone 208.

[0059] The drive control unit 246 operates the arm unit 210A and the moving drive unit 210B based on the motion control information acquired from the information providing server 100. The drive control unit 246 also drives other parts of the robot device 200 (e.g., the head, torso, waist, etc.) based on the motion control information to change the posture, etc. The drive control unit 246 may also drive the robot device 200 based on the basic motion information 262 and customized motion information 264 stored in the storage unit 260. The basic motion information 262 is drive control information for executing a basic motion according to the type, etc. of the robot device 200. The customized motion information 264 is drive control information for causing the robot device 200 to execute a specific motion, which is registered in advance for each operator, for example.

[0060] The output control unit 248 causes the monitor 214 to output an image based on the information acquired from the information providing server 100. In addition, the output control unit 248 causes the speaker 216 to output a voice, a warning sound, or the like based on the information acquired from the information providing server 100.

[0061] The storage unit 260 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, or a RAM, etc. The storage unit 260 stores, for example, basic operation information 262, customized operation information 264, programs, and other information.

[0062] [Operation terminal] 8 is a diagram showing an example of the configuration of an operation terminal 300 according to an embodiment. The operation terminal 300 includes, for example, a communication unit 310, a visual device 320, an operation device 330, an environmental sensor 340, a control device 360, and a storage unit 380. The visual device 320, the operation device 330, and the environmental sensor 340 are examples of an "HMI (Human Machine Interface)."

[0063] The communication unit 310 communicates with the information providing server 100 via the network NW under the control of the control device 360. The communication unit 310 may also communicate with other operation terminals 300 and the robot device 200.

[0064] The visual device 320 may be, for example, an HMD, or a configuration in which a sensor or the like is provided on a glasses-type device or display. The visual device 320 includes, for example, an image display unit 322, a gaze detection unit 324, a sensor 326, a microphone 327, a speaker 328, and a control unit 329. The visual device 320 outputs and displays the state image and sound of the robot device 200 received by the control device 360 ​​from the information providing server 100, and detects the movement of the operator's gaze, etc.

[0065] The image display unit 322 is, for example, an LCD, an organic EL display, etc. Under the control of the control unit 329, the image display unit displays an image output by an output control unit 368, which will be described later.

[0066] The gaze detection unit 324 detects the gaze of an operator using the visual device 320 under the control of the control unit 329, and outputs the detected gaze information (operator sensor value) to the control device 360. The gaze information is, for example, information including a gaze vector.

[0067] The sensor 326 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 326 detects the tilt and rotation of the head of the operator wearing the visual device 320, and outputs the detected head operation information (operator sensor value) to the control device 360.

[0068] The microphone 327 receives voice input from the operator. The speaker 328 outputs voice, warning sounds, and the like output by the output control unit 368.

[0069] The control unit 329 controls the execution of gaze detection by the gaze detection unit 324 based on control information from the control device 360, controls detection by the sensor 326, controls the image display on the image display unit 322, and accepts information input from the operator via the visual device 320.

[0070] The operation device 330 includes, for example, a sensor (operator sensor) 332, a control unit 334, a feedback unit 336, and an input unit 338. The operation device 330 includes, for example, a tactile data glove that is worn on the hand of the operator.

[0071] The sensor 332 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 332 includes a plurality of sensors. The sensor 332 tracks the movement of each finger using, for example, two sensors. The sensor 332 detects operator arm information (operator sensor value) which is information on the orientation and movement of each finger of the operator, the posture and position of the operator's arm such as the movement of the hand, etc., under the control of the control unit 334, for example. The operator arm information (operator sensor value) includes information covering the entire human arm, such as hand tip position and posture information, angle information of each finger, elbow position and posture information, and information tracking the movement of each part.

[0072] The control unit 334 outputs the operator's arm information detected by the sensor 332 to the control device 360. In addition, the control unit 334 controls the feedback unit 336 based on feedback information acquired from the control device 360.

[0073] The feedback unit 336 feeds back feedback information to the operator in response to the control of the control unit 334. The feedback unit 336 feeds back sensations to the operator in response to the feedback information, for example, by means of vibration applying means (not shown), air pressure applying means (not shown), hand movement restricting means (not shown), temperature sensing means (not shown), hardness or softness sensing means (not shown), vibration sensing means (not shown), or the like, which are attached to the arm unit 210A of the robot device 200.

[0074] The input unit 338 is, for example, an input device other than the tactile data glove, such as a keyboard, a mouse, a lever, a touch panel, a microphone, etc. The input unit 338 accepts input of operation details for the robot device 200 from each input device.

[0075] The environment sensor 340 detects, for example, the operation of the operator. The environment sensor 340 includes, for example, a camera (an example of an imaging unit) 342, a sensor 344, and an object position detection unit 346. The camera 342 captures an image including the operator. The camera 342 is, for example, an RGB camera. The camera 342 outputs the captured image to the object position detection unit 346. Note that in the environment sensor 340, the positional relationship between the camera 342 and the sensor 344 is known.

[0076] The sensor 344 is, for example, a depth sensor. The sensor 344 outputs a detection result to the object position detection unit 346. Note that the camera 342 and the sensor 344 may be distance sensors.

[0077] The object position detection unit 346 detects the three-dimensional position, size, shape, and the like of the target object in the captured image by using a known method based on the image captured by the camera 342 and the detection result detected by the sensor 344. The object position detection unit 346 performs image processing (edge ​​detection, binarization processing, feature extraction, image enhancement processing, image extraction, pattern matching processing, and the like) on the image captured by the camera 342 with reference to a pattern matching model and the like stored in the object position detection unit 346 to estimate the position of the object. When multiple objects are detected from the captured image, the object position detection unit 346 detects the position of each object. The object position detection unit 346 transmits the detected object position information to the control device 360. The data transmitted by the environment sensor 340 may be, for example, a point cloud having position information.

[0078] The control device 360 ​​includes, for example, a communication control unit 362, an acquisition unit 364, an operation content generation unit 366, and an output control unit 368. The communication control unit 362, the acquisition unit 364, the operation content generation unit 366, and the output control unit 368 are realized by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuit units) such as an LSI, ASIC, FPGA, or GPU, or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and may be installed by mounting the storage medium in a drive device.

[0079] The communication control unit 362 communicates with the information providing server 100 by wireless communication via the communication unit 310, for example, to transmit and receive information. The communication control unit 362 may also communicate with other operation terminals 300 via the communication unit 310. The communication control unit 362 also communicates with the visual device 320, the operation device 330, and the environmental sensor 340.

[0080] The acquisition unit 364 acquires the division information obtained from the visual device 320, the operation device 330, and the environment sensor 340. The operation content generation unit 366 generates operation content for the robot device 200 based on the information acquired by the acquisition unit 364. For example, the operation content generation unit 366 generates operation content related to line of sight information and head direction based on the information acquired from the visual device 320. The operation content generation unit 366 also generates operation content related to the movement of the arm unit 210A of the robot device 200 based on the information acquired from the operation device 330. The operation content generation unit 366 also generates operation content related to the posture, movement direction, and movement amount of the robot device 200 based on the information acquired from the environment sensor 340. The information generated by the operation content generation unit 366 is transmitted to the information providing server 100 via the communication unit 310.

[0081] The output control unit 368 outputs the provided information acquired from the information providing server 100 to the visual device 320 and the operation device 330. For example, the output control unit 368 causes the image acquired from the information providing server 100 to be output to the image display unit 322 of the visual device 320. The output control unit 368 also causes the speaker 328 to output voice, warning sound, and the like based on the information acquired from the information providing server 100. The output control unit 368 also causes the operation device 330 to output feedback information for conveying tactile information acquired from the information providing server 100 to the operator.

[0082] The storage unit 380 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, or a RAM, etc. The storage unit 380 stores, for example, programs and other information.

[0083] [Regarding remote control of the robot device according to the embodiment] Next, the remote operation of the robot device 200 by the information provision system 1 according to the embodiment will be specifically described. FIG. 9 is a diagram for explaining the remote operation of the robot device 200 according to the embodiment. FIG. 9 shows an example in which two operators U1 and U2 take turns operating one robot device 200. As shown in FIG. 9, each of the operators U1 and U2 is equipped with a visual device 320 and an operation device 330. The operation terminals 300-1 and 300-2 may be installed in different places.

[0084] In addition, the operators U1 and U2 are provided with an environmental sensor 340 for measuring the operator's movements and the surrounding environment. Note that a configuration similar to the environmental sensor 340 may be attached to, for example, the robot device 200. In addition, the operation device 330 includes, for example, an operation device 330a worn on the operator's left hand and an operation device 330b worn on the right hand. The operation contents input from each of the operation devices 330a and 330b are used to control the movements of the left hand LH and right hand RH of the robot device 200. The position of the operator's arm, the direction of the face, and the like are detected by the environmental sensor 340. The detected data is transmitted to the information providing server 100 via a network.

[0085] The information providing server 100 estimates the intention of the operator based on the operation contents obtained from the terminal that operates the robot device 200, among the operation terminals 300-1 and 300-2. Here, an example of information estimated by the intention estimation unit 153 will be specifically described. The intention estimation unit 153 estimates the operator's intention of operation based on the acquired operation contents from the operator. For example, the intention estimation unit 153 classifies the posture of the arm including the gripping part of the robot device by classifying the posture of the arm of the operator based on the operator sensor value of the operation terminal 300. In addition, the intention estimation unit 153 estimates the intention of operation that the operator wants the robot device 200 to perform based on the classification result. The intention estimation unit 153 estimates, for example, the movement of hands and fingers at each time, the purpose of the work that the robot device 200 wants to perform, the work contents, the movement of hands and fingers at each time, and the like as the operator's intention of operation. The purpose of the work is, for example, gripping an object, moving an object, and the like. The work content includes, for example, grasping and lifting an object, grasping and moving an object, and the like.

[0086] Furthermore, the intention estimation unit 153 may estimate the operator's intention to operate by, for example, inputting the operation content into a learned model or the intention estimation information 174 stored in the storage unit 170. In the embodiment, the intention estimation is performed by classifying the grip posture, so that the operator's intention to operate can be estimated with high accuracy. Note that other methods may be used to classify the grip posture.

[0087] The intention estimation unit 153 may also perform an integrated estimation using the gaze and the movement of the arm. In this case, the intention estimation unit 153 may input gaze information, hand movement information, and position information of the object OB on the table TB to a trained model to estimate the operator's intention.

[0088] In the example of Fig. 9, the intention estimation unit 153 estimates an object to be grasped based on, for example, gaze information. Next, the intention estimation unit 153 estimates the posture of the operator's hand based on the estimated object to be grasped. Alternatively, the intention estimation unit 153 may first estimate the posture of the operator's hand based on the operation content, and then estimate the object to be grasped from the estimated posture of the operator's hand. Also, when operations of different parts are assigned to each of the operators U1 and U2, the intention estimation unit 153 may estimate the intention based on the operation content of each of the operators.

[0089] For example, when an object OB is placed on a table TB shown in Fig. 9, the intention estimation unit 153 estimates that the object OB will be grasped based on the posture of the hand. The intention estimation unit 153 may also estimate in advance the future trajectory of the hand intended by the operator based on the operation content and state information of the robot device 200. The intention estimation unit 153 may also estimate the object to be grasped and the position of the object by using the detection result detected by the sensor, the result of image processing of the image captured by the environment sensor 340, and the like.

[0090] In addition, since the coordinate system is different between the environment in which the operator operates and the robot operating environment, for example, calibration between the operator operating environment and the robot operating environment may be performed when starting the robot device 200. In addition, when gripping an object, the information providing server 100 may determine the gripping position based on the gripping force of the robot device and the frictional force between the object and the gripping part, etc., taking into consideration the error in the gripping position at the time of gripping.

[0091] When the robot device 200 is not remotely operated, its operation is controlled according to the control of the control device 240, and when it is remotely operated, its operation is controlled according to operation control information generated by the information providing server 100 based on the estimated intention, etc.

[0092] The control device 240 of the robot device 200 controls the driving unit 210 based on the operation control information from the information providing server 100. The robot device 200 outputs information such as sensor information detected by the sensor 204, an image captured by the camera 206, and sound output by the microphone 208 to the information providing server 100. The information providing server 100 generates information to be provided to the operation terminals 300-1 and 300-2 based on the acquired sensor information, image, sound, and the like. In this case, the information providing server 100 may generate an image in which at least a part of the sensor information is superimposed on the image. The information providing server 100 generates information to be provided including information to be transmitted by the sense of touch. Since the robot device 200 can be operated by a plurality of operators, an image in which information for identifying the operator of each part of the robot device 200 is superimposed on the camera image may be generated. The operation terminal 300 displays the image output from the information providing server 100. This allows, when multiple operators operate the robot device 200, the status of the robots operated by the other operators to be understood.

[0093] Furthermore, when performing remote operation as described above, the operators may feel lonely because they are operating the operation terminals 300 in different locations. Therefore, the information providing server 100 generates information that makes the operator feel (create an illusion, create a simulated experience) that he or she is in the working space where the robot device 200 exists, and provides the generated image to the operation terminal 300.

[0094] For example, when a user instructs to remotely operate the robot device 200, the information providing server 100 transmits information (image) to the operation terminal 300 to select a location in space where the robot device 200 is to work. The location is determined, for example, based on seat information in the space. FIG. 10 is a diagram showing an example of an image IM10 for allowing a user to select a seat. Note that the display mode of the image IM10, such as the layout and display contents, is not limited to the example of FIG. 10. The same applies to the display modes of other images described below. The contents of the image IM10 are generated, for example, by the output control unit 156.

[0095] The image IM10 includes, for example, a character display area A11 and a seat selection area A12. Character information that prompts the user to select a seat is displayed in the character display area A11. In the example of FIG. 10, character information such as "Which seat would you like to work at?" is displayed. The information providing server 100 may generate voice data equivalent to the character information and output the generated data from the operation terminal 300.

[0096] In the seat selection area A12, for example, information on the use status of seats in the work space where the robot device 200 operated by the user is operated is displayed. In the example of FIG. 10, six seats S1 to S6 are displayed corresponding to the positions of the seats in the real space. When the seats are set in the virtual space, the seat positions may be different depending on, for example, a normal business mode, a conference mode, etc. The seats may be arranged in a circle in the conference mode, for example. The seat arrangement may be the same as that of a company, and the seats of employees who are in the office may be displayed as "in office", etc., to mix the real world and the virtual world. The information providing server 100 refers to the use status information 172 managed by the use status management unit 146, and displays the information on the seats in use in the seat selection area A12. In the example of FIG. 10, an image indicating that the seats S2 and S5 are in use and the seats S1, S3, S4, and S6 are vacant is displayed in the seat selection area A12. Furthermore, the information providing server 100 may display information identifying the currently using user in the seat selection area A12. In the example of Fig. 10, it is shown that the seat S2 is used by the user U4, and the seat S5 is used by the user U3. This allows the user to select a desired seat based on the current usage situation. The following description will be given assuming that the user has selected the seat S4.

[0097] When the seat is specified, the information providing server 100 updates the usage status information 172, generates motion control information for moving the robot device 200 to the position of seat S4 by the user, and transmits the generated motion control information to the robot device 200. The robot device 200 moves to the position of S4 based on the received motion control information, and transmits images captured by the camera 206, etc. to the information providing server 100. The information providing server 100 transmits the images obtained from the robot device 200 to the operation terminal 300, and causes the visual device 320 to display them.

[0098] Fig. 11 is a diagram showing an example of an image IM20 captured by the robot device 200. The image IM20 is a diagram of a virtual space viewed to the right from the seat S4. In the example of Fig. 11, a user U3 and another robot device 200 (U4) operated by a user U4 are displayed. This allows the user to recognize nearby workers or other robot devices.

[0099] Also, by looking directly at the image IM20, it is not possible to know who is operating the other robotic device 200 (U4). Furthermore, as remote work using the other robotic devices 200 progresses in the future, there may be cases where there are only a plurality of robotic devices 200 in the vicinity, which may cause a sense of loneliness. Therefore, the information providing server 100 acquires an image representing the user U4 operating the other robotic device 200 (U4) from the image information stored in the user information 171, and generates an image by superimposing the acquired image on the other robotic device 200 (U4).

[0100] FIG. 12 is a diagram for explaining superimposing image information of a user on another robot device 200(U4) existing in the vicinity. In the example of FIG. 12, an image IM30 is generated by superimposing an image IM(U4) of a user U4 operating another robot device 200(U4) on an image IM20. For example, the information providing server 100 detects the orientation of the other robot device 200(U4) included in an image captured by the camera 206 of the robot device 200, and acquires image information captured at an angle corresponding to the detected angle from the user information 171. In addition, the acquired image is enlarged or reduced, and the other robot device 200(U4) is superimposed so as to be hidden according to the size of the other robot device 200(U4) included in the image IM20. In addition, the information providing server 100 partially moves the positions of the arms and legs of the image of the user U4 according to the actions (movements of the arms and legs) of the other robot device 200(U4) and displays information on the actions of the operator. As described above, when the image provided to the operation terminal 300 includes other robotic devices 200, the image representing the user operating the other robotic devices 200 is superimposed and moved in accordance with the action, making the user feel as if the operator is actually operating at that position, and even during remote operation, the user can work while keeping track of the state of the surrounding workers without feeling alone. Also, since information related to the action is displayed, it is possible to distinguish from the image whether the user is busy or not.

[0101] In addition, the information providing server 100 acquires not only images but also sounds such as surrounding conversations as shown in FIG. 12, and provides the acquired sounds to the operation terminal 300. In this case, the information providing server 100 outputs the sound acquired from the operation terminal operating the other robotic device 200 (U4) by localizing the sound image at the location of the other robotic device 200 (U4). In addition, the voice of the worker U3 is inputted by the microphone 208 and outputted. This allows the operator to hear the sound according to the location, so that the operator can feel as if he or she is in the work space more realistically. In addition, the work performed by the other robotic device 200 (U4) or the worker in the real space can be recognized from the image, so that the user can manage his or her own work and progress while directly checking the work situation of the other workers.

[0102] Furthermore, when the line of sight (face direction) of the robot device changes, the information providing server 100 provides the above-mentioned image, sound, and other information to other robot devices seated in the changed direction. This allows the operator of the robot device 200 in the vicinity to be displayed not only when the robot device turns left and right but also when the robot device turns back.

[0103] Also, when images and sounds of other workers are provided during work, the information may be perceived as annoying. Therefore, the information providing server 100 may provide an image from which the workers and robotic devices present in the vicinity have been deleted, when requested by the operation terminal 300. FIG. 13 is a diagram showing an example of an image IM40 from which the images of the workers and robotic devices 200 present in the vicinity have been deleted. In the example of FIG. 13, an image IM40 from which the other robotic devices 200 and workers displayed in the areas A41 and A42 have been deleted is displayed. A background image taken when the other robotic devices 200 and workers are not present may be superimposed on the deleted areas A41 and A42, or another mask image may be displayed. Also, the information providing server 100 may delete only the other robotic devices or only a specific user (worker) in response to a request from a user. Also, in addition to the image, the corresponding sound may be deleted (mute function). This allows the content of the surrounding situation to be switched according to the work situation, etc., and the user can work in his / her preferred environment.

[0104] In the embodiment, when seats are assigned in advance on the virtual section, the location of the operator may be managed in association with the assigned seat information, and an image showing the managed virtual space may be output to the operation terminal. FIG. 14 is a diagram showing an example of arranging information based on information representing the operator and information related to the action on the virtual space. The information providing server 100 generates an image IM50 by superimposing images IM(U3) and IM(U4) representing the operators U3 and U4 on an image showing the seat positions in the virtual space as shown in FIG. 14, at positions corresponding to the positions of the images showing the seats used by the operators. The information providing server 100 also causes the operation terminal 300 to output information in which the voices of the operators are localized as sound images at the positions where the images IM(U3) and IM(U4) are superimposed. This makes it possible to provide information that makes the user feel as if they are in the working space even in the virtual space.

[0105] [Processing sequence] Next, the process executed by the information providing system in the embodiment will be described with reference to a sequence diagram. Fig. 15 is a sequence diagram for explaining the process executed by the information providing system 1 in the embodiment. In the example of Fig. 15, the information providing server 100, the robot device 200, and the operation terminal 300 are used for the description.

[0106] The operation terminal 300 accesses the information providing server 100 to remotely operate the robot device 200 (step S100). The information providing server 100 performs authentication processing to determine whether or not the information providing system 1 is available based on authentication information input from the operation terminal 300 (step S102). The following describes a case where the use is permitted by authentication. If the use is permitted, the information providing server 100 generates an image (e.g., image IM10) for selecting a seat (step S104). The information providing server 100 displays the generated image IM10 on the operation terminal 300 (step S106). The information providing server 100 also acquires seat information selected by the operation terminal 300 (step S108) and manages the usage status (step S110). The information providing server 100 also generates motion control information for moving to the selected seat, and transmits the generated motion control information to the robot device 200 (step S112).

[0107] The robot device 200 moves to the selected seat position based on the operation control information from the information providing server 100 (step S114). The robot device 200 outputs the detection results of the sensors, images captured by the cameras, and the like to the information providing server 100 (step S116).

[0108] The information providing server 100 acquires the information transmitted from the robot device 200, and generates information (provided information) to be provided to the operation terminal 300 based on the acquired information (step S118). Then, the information providing server 100 transmits the generated provided information to the operation terminal 300 (step S120).

[0109] The operation terminal 300 receives the provided information transmitted from the information providing server 100, and provides the information by outputting the received information from the visual device 320, the operation device 330, etc. (step S122). Next, the operation terminal 300 transmits the operation content related to the robot device 200 to the information providing server 100 (step S124).

[0110] The information providing server 100 performs intention estimation based on the operation contents obtained from each of the operation terminals 300 (step S126), generates operation control information based on the intention estimation result (step S128), and transmits the generated operation control information to the robot device 200 (step S130).

[0111] The robot device 200 performs an operation based on the operation control information (step S132), and transmits information obtained from sensors or the like (sensor information), camera images, and the like during or as a result of the operation to the information providing server 100 (step S134).

[0112] The information providing server 100 stores the received information in the storage unit 170 (step S136), generates information to be provided for transmitting the acquired information to the operation terminal 300 (step S138), and transmits the generated information to be provided to the operation terminal 300 (step S140). The operation terminal 300 outputs the acquired information to provide the information to the operator (step S142). After that, the processes of steps S124 to S142 are continuously performed until the operation of the robot device 200 is completed.

[0113] <Modification> A part or all of the processes of the information providing server 100, the robot device 200, and the operation terminal 300 in the embodiment may be realized by AI (Artificial Intelligence) technology. In addition, a part or all of the functional units or information stored in the functional units or storage units included in the information providing server 100, the robot device 200, and the operation terminal 300 may be included in other devices.

[0114] In addition, in the above-described embodiment, information representing the operator and information regarding his / her actions are arranged and output at locations arranged in space, but, for example, when users are arranged in seats in a virtual space in a web conference mode or the like, the arrangement may be changed so that only the designated person is present in front of the user.

[0115] Furthermore, the information providing server 100 may transmit operation history information, intention estimation information, and the like to the robot manufacturer terminal 400. This can be used for developing the robot manufacturer terminal 400 or products, updating programs, and the like. In this case, a discount or reduction service for system usage may be provided to a company that performs eco-driving of the robot device 200 (for example, low-power driving, low-load driving, low-communication-amount driving), or a company that provides a robot control method suitable for an operator or telework characteristics. This can encourage users to provide information, which can be useful for improving services.

[0116] In the above embodiment, when a single robot device 200 is operated by multiple operators, the operation intentions and work statuses of each operator are displayed. However, for example, the above-mentioned intention estimation result, work status, and information about the operator may be provided between multiple robot devices 200 that exist within a predetermined distance or between robot devices performing a joint task. By using the intention estimation function between robots, it becomes possible to share consciousness more effectively than between humans. In addition, since the statuses of surrounding robot devices 200 can be acquired, group work, etc. can be performed smoothly.

[0117] The embodiment described above includes an information providing server 100 that includes a communication unit 110 that communicates with one or more operation terminals 300 via a network NW, an acquisition unit 130 that acquires information related to the behavior of the operator of the operation terminal 300, a management unit 140 that manages the operator's location in space, and an output control unit 156 that arranges information based on information representing the operator and information related to his / her behavior at the operator's location in space managed by the management unit 140 and outputs the arranged information to the operation terminal 300.By doing so, it is possible to provide information that makes the user feel as if they are in the work space even when operating remotely.

[0118] Specifically, according to the above-mentioned embodiment, for example, the position (location) of the operator in the virtual space (workplace) is coordinated, and information showing the state of the operator working in the virtual space based on the position is arranged, so that the state seen from the operator's own position in the virtual space (the state of people working in the same space) can be output to the image seen by the operator (image of the visual device). Furthermore, according to the above-mentioned embodiment, when performing teleworking by remotely operating a robot device, various devices can be used more generally and can be operated by a team. Furthermore, in the past, only limited work could be performed with limited dedicated robots even when remotely operated, but according to the present embodiment, anyone can easily perform teleworking with a highly versatile robot device.

[0119] According to the embodiment, robot information and sensor information (information that may include corporate secrets) can be managed separately, and a server capable of handling confidential information can be provided. Remote operators can access the same server and can share the control rights and sensor information of the same robot or different robots. For example, when operating the same robot device, multiple operators can operate different arms, so that a senior and a junior can simultaneously operate the same arm of the same robot, and can add and subtract forces while sharing sensor information such as touch. When operating different robots, telepathy is possible by sharing an intention estimation function and disclosing the intention of others. According to the embodiment, one operator can operate two robots. In this case, for example, when a large desk is to be carried by two robots, one robot is brought to the end of the desk in advance and synchronized with the other robot. When the other robot is brought to the other end and an operation to lift it is performed, the robots are lifted together by synchronization processing. Furthermore, in the embodiment, more accurate work can be performed than when humans call out to each other to perform the operation. In addition, in the embodiment, when one operator causes multiple robots to perform the same work, the supported teleoperation function can absorb individual differences between robots and individual differences between work objects.

[0120] The above-described embodiment can be expressed as follows. A server computer that communicates with a mobile object and a mobile terminal of a user of the mobile object, A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, Communicating with one or more operation terminals via a network; Acquire information regarding the behavior of an operator of the operation terminal; Managing the location of the operator in space; arranging information based on information representing the operator and information related to the action at a location in the space of the managed operator, and outputting the arranged information to the operation terminal; An information providing server configured to:

[0121] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0122] 1...information providing system, 100...information providing server, 110, 202, 310...communication unit, 120...communication control unit, 130...acquisition unit, 140...management unit, 150...control unit, 170...storage unit, 200...robot device, 204...sensor, 206...camera, 208...microphone, 210...drive unit, 214...monitor, 216...speaker, 240...control device, 260...storage unit, 300...operation terminal, 320...visual device, 330...operation device, 340...environmental sensor, 360...control device, 380...storage unit

Claims

1. A communication unit that communicates with a plurality of operation terminals via a network; an acquisition unit that acquires information regarding an action of an operator of a first operation terminal among the plurality of operation terminals and information regarding an action of an operator of an operation terminal other than the first operation terminal; a management unit that manages the location of the operator of the other operation terminal in a virtual space of an image captured by an imaging unit of a space in which one or more robot devices exist; an output control unit that arranges information based on information representing the operator of the other operating terminal and information regarding the behavior at an arrangement location in a virtual space of the operator of the other operating terminal managed by the management unit, and outputs the arranged information to the first operating terminal; the information about the actions of the operators of the other operation terminals includes operation content for remotely operating the one or more robot devices; the information based on the information representing the operator of the other operating terminal and the information relating to the behavior includes information in which an image representing the operator of the other operating terminal is superimposed at a position where a robot device operated by the operator of the other operating terminal is present among the one or more robot devices present in the image captured by the imaging unit, and information in which a sound image is localized for a voice of the operator of the other operating terminal, the output control unit, in response to a request from the first operation terminal, generates an image by deleting at least one of an image representing an operator of the other operation terminal and an image representing the robot device from an image captured by the imaging unit, and provides the image to the first operation terminal. Information server.

2. the management unit generates an image that allows an operator of the first operation terminal to select a location, transmits the generated image to the first operation terminal, and manages the location of the operator of the first operation terminal based on the location selected using the image.

2. The information providing server according to claim 1.

3. the plurality of operation terminals include a terminal for remotely operating one or more robot devices, 3. The information providing server according to claim 1 or 2.

4. the output control unit acquires a voice of an operator of the other operation terminal, and outputs the acquired voice by localizing a sound image at a location of the operator of the other operation terminal.

4. The information providing server according to claim 1.

5. An information providing server according to any one of claims 1 to 4; The plurality of operation terminals; one or more robot devices remotely operated by the plurality of operation terminals; An information provision system comprising:

6. The computer Communicate with multiple operation terminals via a network, acquiring information on an action of an operator of a first operation terminal among the plurality of operation terminals and information on an action of an operator of another operation terminal other than the first operation terminal; managing locations of operators of the other operation terminals in a virtual space of an image captured by an imaging unit of a space in which one or more robot devices exist; arranging information representing the operator of the other operation terminal and information related to the behavior at a location in a virtual space of the operator of the other operation terminal that is managed, and outputting the arranged information to the first operation terminal; the information about the actions of the operators of the other operation terminals includes operation content for remotely operating the one or more robot devices; the information based on the information representing the operator of the other operating terminal and the information relating to the behavior includes information in which an image representing the operator of the other operating terminal is superimposed at a position where a robot device operated by the operator of the other operating terminal is present among the one or more robot devices present in the image captured by the imaging unit, and information in which a sound image is localized for a voice of the operator of the other operating terminal, generating an image obtained by deleting at least one of an image representing the operator of the other operation terminal and an image representing the robot device from an image captured by the imaging unit in response to a request from the first operation terminal, and providing the image to the first operation terminal; Information provision method.

7. On the computer, Communicating with multiple operation terminals via a network, acquiring information on an action of an operator of a first operation terminal among the plurality of operation terminals and information on an action of an operator of an operation terminal other than the first operation terminal; managing locations of operators of the other operation terminals in a virtual space of an image captured by an imaging unit of a space in which one or more robot devices exist; arranging information representing the operator of the other operation terminal and information related to the behavior at a location in a virtual space of the operator of the other operation terminal being managed, and outputting the arranged information to the first operation terminal; the information about the actions of the operators of the other operation terminals includes operation content for remotely operating the one or more robot devices; the information based on the information representing the operator of the other operating terminal and the information relating to the behavior includes information in which an image representing the operator of the other operating terminal is superimposed at a position where a robot device operated by the operator of the other operating terminal is present among the one or more robot devices present in the image captured by the imaging unit, and information in which a sound image is localized for a voice of the operator of the other operating terminal, generating an image obtained by deleting at least one of an image representing an operator of the other operation terminal and an image representing the robot device from an image captured by the imaging unit in response to a request from the first operation terminal, and providing the image to the first operation terminal; program.

Images