ROBOT, ROBOT CONTROL METHOD, AND COMPUTER PROGRAM

Abstract

To achieve natural and continuous communication between users located in different locations from each other.SOLUTION: A robot includes a display, a camera, a communication interface, a movement mechanism, and a control unit. The control unit includes a movement control unit, a remote video processing unit, and an on-site video processing unit. The movement control unit controls the movement mechanism to move the robot. The remote video processing unit acquires a remote video including a video representing a plurality of remote users located at locations different from the current location of the robot from an external network via the communication interface, and displays the acquired remote video on the display. The on-site video processing unit transmits an on-site video which is a video taken by the camera, to the external network via the communication interface, in order to cause the plurality of remote users to visually recognize the on-site video.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology disclosed in this specification relates to a robot, a robot control method, and a computer program. [Background technology]

[0002] There is a known technology called telexistence (a remote sense of presence or remote presence) that uses a robot as an avatar of a human located at a different location from the robot. By using this technology, a person at a certain location (remote location) can possess (take possession of) a robot located at another location (local location), thereby real-time communication and / or interaction (hereinafter simply referred to as "communication") can be realized between the person at the remote location and another person at the local location (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-23464 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional telexistence technology involves a person in a remote location transferring into a robot in the local area, which can easily lead to psychological gaps in communication between the people in the remote location and the people in the local area. For example, the people in the local area tend to think, "They've come all the way from a remote location, so I have to pay them attention," and this mentality is conveyed to the person in the remote location, placing a psychological burden on them and hindering natural and continuous communication between the two.

[0005] As described above, conventional technologies have room for improvement with respect to natural and continuous communication between different locations using telexistence technology.

[0006] This specification discloses a technique that can solve the above-mentioned problems. [Means for solving the problem]

[0007] The technology disclosed in this specification can be realized, for example, in the following forms.

[0008] (1) The robot disclosed in this specification includes a display, a camera, a communication interface, a movement mechanism for moving the robot, and a control unit. The control unit has a movement control unit, a remote image processing unit, and a local image processing unit. The movement control unit controls the movement mechanism to move the robot. The remote image processing unit acquires remote images including images of multiple remote users located at locations different from the current location of the robot from an external network via the communication interface, and displays the acquired remote images on the display. The local image processing unit transmits the local images, which are images captured by the camera, to the external network via the communication interface so that the multiple remote users can view them.

[0009] In this manner, in the present robot, the movement control unit controls the movement mechanism to move the robot, the remote image processing unit displays on the display a remote image including an image showing the multiple remote users, and the local image processing unit allows the multiple remote users to view the local image, which is an image captured by a camera. Therefore, the present robot functions as a collective telexistence device that can have multiple remote users possess the robot. Therefore, the relationship between the remote user and the local user is not an individual-to-group relationship but a group-to-group relationship, and the psychological burden of the remote user can be reduced. Furthermore, the multiple remote users who possess the present robot share an experience as if they were riding together in one robot, and a sense of closeness is fostered between the multiple remote users. From the above, the present robot can realize natural and continuous communication between users located at different locations.

[0010] (2) In the above robot, the control unit may further include an information space construction unit that constructs an information space on the external network shared by the multiple remote users, and the local image processing unit may project the local image onto the information space to allow the multiple remote users virtually located in the information space to view the local image. By adopting this configuration, the multiple remote users do not need to actually gather at one location, and the multiple remote users located at different locations can virtually gather in one information space and communicate with the local users.

[0011] (3) In the robot, the remote video may include avatar videos of the multiple remote users. By adopting this configuration, the remote users do not need to use a device with a camera function, and communication using more flexible and diverse video expressions can be realized.

[0012] (4) In the robot, the display may be a 360-degree display. By adopting this configuration, it is possible to realize communication between users located at different locations via more realistic images.

[0013] (5) In the robot, the camera may be a 360-degree camera. By adopting this configuration, it is possible to realize communication between users located at different locations via more realistic images.

[0014] (6) The robot may further include a 360-degree microphone, and the control unit may further include a local voice processing unit that transmits local voice, which is voice captured by the 360-degree microphone, to the external network via the communication interface so that the remote users can hear the local voice and recognize the direction of the sound source. By adopting this configuration, more realistic voice-based communication can be realized between users located at different locations.

[0015] (7) The robot may further include a directional speaker, and the control unit may further include a remote voice processing unit that acquires remote voices emitted from the multiple remote users from the external network via the communication interface and outputs the acquired remote voices from the directional speaker so that the directions in which the remote users are located can be recognized. By employing this configuration, more realistic communication via voices can be realized between users located at different points.

[0016] (8) The robot may further include a robot arm, and the control unit may further include a robot arm control unit that receives operation instructions from the multiple remote users from the external network via the communication interface and operates the robot arm in response to the received operation instructions. By employing this configuration, communication (interaction) via the robot arm can be realized between users located at different locations.

[0017] The technology disclosed in this specification can be realized in various forms, such as a robot, a robot control device, a robot system including a robot and a robot control device, a robot control method, a computer program for realizing these methods, a non-transitory recording medium on which the computer program is recorded, etc. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is an explanatory diagram illustrating a schematic configuration of a remote communication system 10 according to a first embodiment. [Diagram 2] FIG. 1 is a perspective view showing the external configuration of a robot 100. [Diagram 3] A block diagram showing the functional configuration of the robot 100. [Figure 4] Block diagram showing the functional configuration of the HMD 200 [Diagram 5]FIG. 1 is an explanatory diagram showing the flow of a remote communication process executed in the remote communication system 10 of the first embodiment. [Figure 6] An explanatory diagram showing the configuration of the conventional robot 100X [Figure 7] FIG. 13 is an explanatory diagram illustrating a schematic configuration of a remote communication system 10a according to a second embodiment. [Figure 8] FIG. 11 is an explanatory diagram showing the flow of a remote communication process executed in a remote communication system 10a according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] A. First embodiment: A-1. Configuration of remote communication system 10: 1 is an explanatory diagram illustrating a schematic configuration of a remote communication system 10 according to a first embodiment. The remote communication system 10 of the present embodiment is a system for realizing real-time communication between users located at different locations using telexistence technology.

[0020] In the example of FIG. 1, the remote communication system 10 realizes communication between five users U located at any of four different points P. In the following description, for convenience, a robot 100 described later is considered as a reference, and among the four different points P, the point P where the robot 100 is located is referred to as a local point Ps, and the other three points P are referred to as a first remote point Pr1, a second remote point Pr2, and a third remote point Pr3, respectively. Hereinafter, the first remote point Pr1, the second remote point Pr2, and the third remote point Pr3 are collectively referred to as a "remote point Pr." For example, if the first remote point Pr1 is considered as a reference, the local point Ps corresponds to a "remote point."

[0021] Two users U (hereinafter referred to as "local users Us") are located at the locality Ps. Meanwhile, one user U (hereinafter referred to as "first remote user Ur1") is located at the first remoteity Pr1, one user U (hereinafter referred to as "second remote user Ur2") is located at the second remoteity Pr2, and one user U (hereinafter referred to as "third remote user Ur3") is located at the third remoteity Pr3. Hereinafter, the first remote user Ur1, the second remote user Ur2, and the third remote user Ur3 are collectively referred to as "remote users Ur". In the example of FIG. 1, one user U is located at each remoteity Pr, but multiple users U may be located at any of the remoteities Pr.

[0022] The remote communication system 10 includes a robot 100 and a head mounted display (hereinafter referred to as "HMD") 200. The devices constituting the remote communication system 10 are communicably connected to each other via an external network NET such as the Internet. In the example of FIG. 1, the remote communication system 10 includes one robot 100 and three HMDs 200. One robot 100 is located at a locality Ps and faces a local user Us. The three HMDs 200 are respectively worn on the heads of three remote users Ur located at a remote location Pr.

[0023] (Configuration of robot 100) Fig. 2 is a perspective view showing the external configuration of the robot 100, and Fig. 3 is a block diagram showing the functional configuration of the robot 100. The robot 100 is a device for realizing communication between different locations by allowing a user U to transfer (become) possessed by the robot 100 using telexistence technology. As will be described later, the robot 100 functions as a collective telexistence device that can allow multiple users U in remote locations Pr to transfer into the robot 100.

[0024] 2 and 3, the robot 100 includes a display 151, a camera 152, a microphone 153, a speaker 154, a robot arm 155, a moving mechanism 156, a communication interface 130, an operation input unit 140, a control unit 110, and a storage unit 120. These units are connected to each other via a bus 190 so as to be able to communicate with each other.

[0025] The display 151 of the robot 100 is a device that displays various images based on digital image data, and is configured, for example, with a liquid crystal display or an organic EL display. In this embodiment, the display 151 is a substantially spherical display, and substantially the entire outer circumferential surface of the sphere is used as the display surface.

[0026] The camera 152 of the robot 100 is a device that generates digital video data by capturing images using an image sensor. In this embodiment, the camera 152 is a 360-degree camera that can generate 360-degree spherical images. In this specification, the term "360 degrees" is not necessarily limited to strict 360 degrees, but means approximately 360 degrees. The camera 152 is installed at a position above the display 151. The camera 152 is preferably capable of generating digital video data with a high resolution, such as 8K or 16K.

[0027] The microphone 153 of the robot 100 is a device that generates digital voice data based on an input voice. In this embodiment, the microphone 153 is a 360-degree surround microphone that can collect sound from 360 degrees around the microphone 153. The microphone 153 is installed at a position above the display 151 and the camera 152.

[0028] The speaker 154 of the robot 100 is a device that reproduces sound based on digital sound data. In this embodiment, a plurality of directional speakers 154 are installed above the display 151 and aligned at approximately equal intervals in the circumferential direction.

[0029] The robot arm 155 of the robot 100 is a mechanical arm capable of performing actions such as grabbing, releasing, and carrying an object. In this embodiment, a plurality of robot arms 155 are installed below the display 151 so as to be aligned at approximately equal intervals along the circumferential direction.

[0030] The movement mechanism 156 of the robot 100 constitutes the lowest part of the robot 100 and moves the robot 100. That is, the movement mechanism 156 has wheels 157 and a drive unit (not shown) that drives the wheels 157, and moves the robot 100 according to, for example, the operation of a remote user Ur and / or a local user Us. In this embodiment, the movement mechanism 156 also has sensors (e.g., LiDAR, radar, far-infrared camera, ultrasonic sensor, etc.) (not shown), and can also move the robot 100 autonomously without human operation.

[0031] The communication interface 130 of the robot 100 is an interface that communicates with other devices on the external network NET using a predetermined communication method. It is preferable that the communication interface 130 is capable of performing communication in accordance with a next-generation mobile communication system such as B5G or 6G. The operation input unit 140 of the robot 100 is composed of, for example, a touch panel, buttons, a keyboard, a microphone, etc., and accepts operations and instructions from an administrator.

[0032] The storage unit 120 of the robot 100 is composed of, for example, a ROM, a RAM, a HDD, an SSD, etc., and is used to store various programs and data, and as a work area when executing various programs and a temporary storage area for data. For example, the storage unit 120 stores a robot control program CP for controlling the robot 100. The robot control program CP is provided in a state stored in a computer-readable recording medium (not shown), such as a CD-ROM, a DVD-ROM, or a USB memory, or is provided in a state in which it can be acquired from an external device (a server or other terminal device on an external network NET) via the communication interface 130, and is stored in the storage unit 120 in a state in which it can operate on the robot 100.

[0033] The control unit 110 of the robot 100 is configured with, for example, a CPU, and controls the operation of each part of the robot 100 by executing a computer program read from the storage unit 120. For example, the control unit 110 functions as a robot operation control unit 111 for controlling the operation of each part of the robot 100 by reading a robot control program CP from the storage unit 120 and executing it. The robot operation control unit 111 includes a remote image processing unit 112, a local image processing unit 113, a remote audio processing unit 114, a local audio processing unit 115, a movement control unit 116, a robot arm control unit 117, and an information space construction unit 118. The functions of each of these units will be described in detail later.

[0034] (HMD200 Configuration) As shown in FIG. 1, the HMD 200 is a device that allows the user U to view an image while being worn on the head of the user U. In this embodiment, the HMD 200 is a non-transparent HMD that completely covers both eyes of the user U, and provides the user U with a virtual reality (VR) experience. The HMD 200 is also capable of inputting and outputting audio, and provides the user U with a VR experience via vision and hearing. Furthermore, the HMD 200 may be capable of providing the user U with a VR experience via other senses (e.g., touch).

[0035] 4 is a block diagram showing the functional configuration of the HMD 200. The HMD 200 includes a right eye display execution unit 251, a left eye display execution unit 252, a microphone 253, a speaker 254, a head movement detection unit 255, a communication interface 230, an operation input unit 240, a control unit 210, and a storage unit 220. These units are connected to each other via a bus 290 so as to be able to communicate with each other.

[0036] The right-eye display execution unit 251 of the HMD 200 has, for example, a light source, a display element (such as a digital mirror device (DMD) or a liquid crystal panel), and an optical system, and generates light representing a right-eye image and guides it to the right eye of the user U, thereby allowing the right eye of the user U to view the right-eye image. The left-eye display execution unit 252 is provided independently of the right-eye display execution unit 251, and like the right-eye display execution unit 251, has, for example, a light source, a display element, and an optical system, and generates light representing a left-eye image and guides it to the left eye of the user U, thereby allowing the left eye of the user U to view the left eye image. In a state in which the right eye of the user U views the right-eye image and the left eye of the user U views the left-eye image, the user U views a 3D image. It is preferable that the right-eye display execution unit 251 and the left-eye display execution unit 252 are capable of reproducing high-resolution digital image data, such as 8K or 16K.

[0037] The microphone 253 of the HMD 200 is a device that generates digital audio data based on input audio. The speaker 254 of the HMD 200 is a device that reproduces audio based on the digital audio data. In this embodiment, the speaker 254 is a directional speaker.

[0038] The head movement detection unit 255 of the HMD 200 is a sensor that detects the movement of the HMD 200 (i.e., the movement of the head of the user U) in order to realize a so-called head tracking function. The movement of the head of the user U is a concept that includes a change in the position and the change in the orientation of the head of the user U. Based on the movement of the HMD 200 detected by the head movement detection unit 255, the right eye display execution unit 251 and the left eye display execution unit 252 switch the images that the user U views, whereby the user U views a VR image that changes naturally according to the movement of the head.

[0039] The communication interface 230 of the HMD 200 is an interface that communicates with other devices on the external network NET using a predetermined communication method. The communication interface 230 is preferably capable of executing communication in accordance with a next-generation mobile communication system such as B5G or 6G. The operation input unit 240 of the HMD 200 is configured with, for example, a touch panel, buttons, etc., and receives operations and instructions from the user U. The operation input unit 240 may be disposed inside the housing of the HMD 200 (the part worn on the head of the user U), or may be configured as a separate unit connected to the housing via a signal line.

[0040] The storage unit 220 of the HMD 200 is composed of, for example, a ROM, a RAM, an SSD, etc., and is used to store various programs and data, and as a work area for executing various programs and a temporary storage area for data. The control unit 210 of the HMD 200 is composed of, for example, a CPU, etc., and controls the operation of each part of the HMD 200 by executing a computer program read from the storage unit 220.

[0041] A-2. Remote Communication Processing: Next, a remote communication process executed in the remote communication system 10 of the first embodiment will be described. The remote communication process is a process for realizing real-time communication between users U located at different locations via vision and / or hearing. Fig. 5 is an explanatory diagram showing the flow of the remote communication process executed in the remote communication system 10 of the first embodiment.

[0042] First, the robot operation control unit 111 (FIG. 3) of the robot 100 connects to the external network NET via the communication interface 130 (S102), and the information space construction unit 118 constructs an information space VS on the external network NET (S104). As shown in FIG. 1, the information space VS is a three-dimensional virtual space such as a metaverse, and in this embodiment, it is a space that imitates the internal space of the robot 100. The information space VS is constructed, for example, on a server (not shown) on the external network NET.

[0043] Further, the control unit 210 of each HMD 200 connects to the external network NET via the communication interface 230 (S202) and accesses the information space VS on the external network NET (S204). As a result, as shown in FIG. 1, each remote user Ur (first remote user Ur1, second remote user Ur2, and third remote user Ur3) wearing the HMD 200 is virtually located in the information space VS as an avatar (first remote user avatar Ura1, second remote user avatar Ura2, and third remote user avatar Ura3). As described above, since the information space VS is a space that imitates the internal space of the robot 100, each remote user Ur virtually located in the information space VS transfers (possesses) to the robot 100 via the information space VS. That is, the multiple remote users Ur are in a state as if they were riding on one robot 100.

[0044] The remote image processing unit 112 (FIG. 3) of the robot 100 acquires a remote image Ir, which is an image showing the state of the information space VS, via the communication interface 130, and displays the acquired remote image Ir on the display 151 (S106). Since a plurality of remote users Ur are virtually located in the information space VS, the remote image Ir includes an image showing the plurality of remote users Ur (more specifically, an image of the avatar Ura of the remote user Ur). As shown in FIG. 1, by displaying the remote image Ir on the display 151 of the robot 100, each local user Us at the local Ps can visually recognize the remote image Ir including an image of the avatar Ura of each remote user Ur. That is, each local user Us at the local Ps can recognize that a plurality of remote users Ur have taken over the robot 100.

[0045] In addition, the local image processing unit 113 (FIG. 3) of the robot 100 generates a local image Is by taking pictures using the camera 152 (S108), transmits the generated local image Is to the information space VS via the communication interface 130 (S110), and projects the local image Is into the information space VS (S302). As a result, each remote user Ur visually recognizes the local image Is projected into the information space VS via the HMD 200 (S206). As shown in FIG. 1, for example, the first remote user Ur1 visually recognizes the scenery around the robot 100 as the local image Is while being transferred to the robot 100 via the information space VS. The first remote user Ur1 also visually recognizes the avatars Ura (the second remote user avatar Ura2 and the third remote user avatar Ura3) of other remote users Ur virtually located in the information space VS.

[0046] Although not shown in FIG. 5, the local voice processing unit 115 (FIG. 3) of the robot 100 generates voice of the local place Ps including the voice of each local user Us (hereinafter referred to as "local voice") using the microphone 153, and transmits the generated local voice to the information space VS via the communication interface 130. As a result, the local voice is reproduced by the speaker 254 of each HMD 200 so that the direction of the sound source can be recognized. In addition, the remote voice processing unit 114 (FIG. 3) of the robot 100 acquires the voice of the remote place Pr including the voice of each remote user Ur generated by the microphone 253 of each HMD 200 (hereinafter referred to as "remote voice") via the communication interface 130, and reproduces the acquired remote voice by the speaker 154 so that the direction of the sound source (in the example of FIG. 1, the direction of the avatar Ura of each remote user Ur) can be recognized.

[0047] In this state, two local users Us are actually located at the local site Ps, and three remote users Ur are virtually located at the local site Ps by transferring to the robot 100 via the information space VS. Therefore, five users U who are actually located at different locations gather virtually at the local site Ps and can communicate with each other through vision and hearing. Note that since multiple remote users Ur share one information space VS, communication between the remote users Ur can also be naturally performed.

[0048] Furthermore, the robot arm control unit 117 (FIG. 3) of the robot 100 monitors whether or not there is a robot arm operation instruction (S112), and when there is a robot arm operation instruction (S112: YES), operates the robot arm 155 according to the instruction (S114). In this embodiment, the remote user Ur can issue a robot arm operation instruction via the operation input unit 240 of the HMD 200. This allows the remote user Ur to communicate with the local user Us (e.g., shake hands or high-five) via the robot arm 155, or perform some kind of operation on an object at the local Ps.

[0049] Further, the movement control unit 116 (FIG. 3) of the robot 100 monitors whether the movement condition is satisfied (S116), and when the movement condition is satisfied (S116: YES), the movement mechanism 156 is operated to move the robot 100 (S118). In this embodiment, the remote user Ur can issue a movement instruction for the robot 100 via the operation input unit 240 of the HMD 200, and the above-mentioned movement condition includes the presence of the movement instruction. Therefore, the remote user Ur can move the robot 100, which is the transfer destination, by issuing the movement instruction, and thereby can virtually move himself / herself in the locality Ps. Naturally, the local image Is visually recognized by the remote user Ur also changes with such virtual movement of the remote user Ur. Note that the movement instruction by the remote user Ur may be issued in response to the head movement detection unit 255 of the HMD 200 detecting the movement of the remote user Ur as the remote user Ur actually moves in the remote place Pr.

[0050] As the above-mentioned movement condition, in addition to or instead of the fact that a movement instruction has been given by the remote user Ur, other conditions may be adopted. For example, as the above-mentioned movement condition, the fact that a movement instruction has been given by the local user Us may be adopted. The movement instruction by the local user Us can be issued via the operation input unit 140 of the robot 100, or via a terminal device (not shown) capable of communicating with the robot 100. Also, as the above-mentioned movement condition, the fact that the distance between the local user Us and the robot 100 has changed due to the movement of the local user Us may be adopted. In this way, the robot 100 can be made to follow the local user Us.

[0051] In the remote communication system 10, the above-mentioned process is repeated and communication between users U located at different locations is continuously performed unless an end instruction (S120, S304, S208) is given in each device. When an end instruction is given in each device (S120: YES, S304: YES, S208: YES), the remote communication process by the remote communication system 10 ends.

[0052] A-3. Advantages of the first embodiment: As described above, the robot 100 constituting the remote communication system 10 of the first embodiment includes the display 151, the camera 152, the communication interface 130, the moving mechanism 156 for moving the robot 100, and the control unit 110. The control unit 110 includes a moving control unit 116, a remote image processing unit 112, and a local image processing unit 113. The moving control unit 116 controls the moving mechanism 156 to move the robot 100. The remote image processing unit 112 acquires a remote image Ir including an image representing a plurality of remote users Ur located at a point different from the current location of the robot 100 from the external network NET via the communication interface 130, and displays the acquired remote image Ir on the display 151. The local image processing unit 113 transmits the local image Is, which is an image captured by the camera 152, to the external network NET via the communication interface 130 so that the plurality of remote users Ur can view it.

[0053] In this manner, in the robot 100 of this embodiment, the movement control unit 116 controls the movement mechanism 156 to move the robot 100, the remote image processing unit 112 displays the remote image Ir including an image representing the multiple remote users Ur on the display 151, and the local image processing unit 113 makes the multiple remote users Ur visually recognize the local image Is, which is an image captured by the camera 152. Therefore, the robot 100 functions as a collective telexistence device that can transfer the multiple remote users Ur to the robot 100. Therefore, the robot 100 can realize real-time communication between the multiple remote users Ur who have transferred to the robot 100 and the local user Us who is actually located at the current location (local area Ps) of the robot 100.

[0054] Here, as shown in FIG. 6, even with a conventional robot 100X (teleexistence device) capable of transferring only one remote user Ur to itself, it is possible to realize real-time communication between one remote user Ur who has transferred to the robot 100X and a local user Us who is actually located at the current location (local Ps) of the robot 100. However, the above communication by the conventional robot 100X is likely to cause a psychological gap. For example, the local user Us is likely to have a mentality of "I have to interact with the local user Us since he has come all the way from a remote location," and such a mentality is conveyed to the remote user Ur, causing a psychological burden and possibly hindering natural and continuous communication between the two.

[0055] In contrast, the robot 100 of this embodiment can have multiple remote users Ur possess it. Therefore, the relationship between the remote user Ur and the local user Us is not an individual-to-group relationship but a group-to-group relationship, and the psychological burden of the remote user Ur can be reduced. Furthermore, the multiple remote users Ur who possess the robot 100 share an experience as if they were riding together on one robot 100, which fosters a sense of closeness between the multiple remote users Ur. From the above, the robot 100 of this embodiment can realize natural and continuous communication between the multiple remote users Ur and the local user Us.

[0056] In this embodiment, the control unit 110 of the robot 100 further includes an information space construction unit 118 that constructs an information space VS shared by multiple remote users Ur on the external network NET, and the local image processing unit 113 projects the local image Is onto the information space VS, thereby allowing multiple remote users Ur virtually located in the information space VS to view it. Therefore, according to the robot 100 of this embodiment, multiple remote users Ur do not need to actually gather at one location, and multiple remote users Ur located at different locations can virtually gather in one information space VS and communicate with local users Us.

[0057] In this embodiment, the remote video Ir includes videos of avatars Ura of multiple remote users Ur. Therefore, according to the robot 100 of this embodiment, the remote users Ur do not need to use a device with a camera function, and communication using more flexible and diverse video expressions can be realized.

[0058] In addition, in this embodiment, the display 151 of the robot 100 is a 360-degree display. Therefore, according to the robot 100 of this embodiment, communication between a plurality of remote users Ur and a local user Us through a more realistic image can be realized.

[0059] In the present embodiment, the camera 152 of the robot 100 is a 360-degree camera. Therefore, according to the robot 100 of the present embodiment, communication between a plurality of remote users Ur and a local user Us through more realistic images can be realized.

[0060] In this embodiment, the robot 100 further includes a microphone 153 that is a 360-degree microphone, and the control unit 110 further includes a local voice processing unit 115 that transmits local voice, which is voice acquired by the microphone 153, to an external network NET via the communication interface 130 so that the multiple remote users Ur can hear the local voice and recognize the direction of the sound source. Therefore, according to the robot 100 of this embodiment, it is possible to realize communication between the multiple remote users Ur and the local user Us via voice with a more realistic feel.

[0061] Moreover, in this embodiment, the robot 100 further includes a directional speaker 154, and the control unit 110 further includes a remote voice processing unit 114 that acquires remote voices, which are voices emitted from a plurality of remote users Ur, from the external network NET via the communication interface 130 and outputs the acquired remote voices from the speaker 154 so that the direction in which each remote user Ur is located can be recognized. Therefore, according to the robot 100 of this embodiment, it is possible to realize communication between a plurality of remote users Ur and a local user Us via voices with a more realistic feel.

[0062] In this embodiment, the robot 100 further includes a robot arm 155, and the control unit 110 further includes a robot arm control unit 117 that acquires operation instructions from a plurality of remote users Ur from the external network NET via the communication interface 130 and operates the robot arm 155 in accordance with the acquired operation instructions. Therefore, according to the robot 100 of this embodiment, communication (interaction) via the robot arm 155 can be realized between a plurality of remote users Ur and a local user Us.

[0063] B. Second embodiment: Fig. 7 is an explanatory diagram showing a schematic configuration of a remote communication system 10a in the second embodiment, and Fig. 8 is an explanatory diagram showing a flow of a remote communication process executed in the remote communication system 10a in the second embodiment. In the following, among the configuration of the remote communication system 10a in the second embodiment and the contents of the remote communication process, the same configuration and processing contents as those in the above-mentioned first embodiment are appropriately omitted by assigning the same reference numerals.

[0064] The remote communication system 10a of the second embodiment includes two robots 100. The two robots 100 are communicably connected to each other via an external network NET. The configuration of each robot 100 is similar to the configuration of the robot 100 in the first embodiment.

[0065] One of the two robots 100 (hereinafter referred to as the "first robot 100(1)") is located at a certain point P (hereinafter referred to as the "first point P(1)"), and the other of the two robots 100 (hereinafter referred to as the "second robot 100(2)") is located at a point P (hereinafter referred to as the "second point P(2)") different from the first point P(1). If the first robot 100(1) is considered as the reference, the first point P(1) where the first robot 100(1) is located is the "local location", and the second point P(2) where the second robot 100(2) is located is the "remote location". Conversely, if the second robot 100(2) is considered as the reference, the second point P(2) where the second robot 100(2) is located is the "local location", and the first point P(1) where the first robot 100(1) is located is the "remote location".

[0066] A plurality of users U (first users U(1) and second users U(2)) (two in the example of FIG. 7) are located at each of the first point P(1) and the second point P(2), and face the robot 100 located at each point P. The remote communication system 10a of the second embodiment realizes real-time communication between a plurality of first users U(1) at the first point P(1) and a plurality of second users U(2) at the second point P(2).

[0067] As shown in FIG. 8, in the remote communication process executed in the remote communication system 10a, the robot operation control unit 111 (FIG. 3) of the first robot 100(1) connects to the external network NET via the communication interface 130 (S102). In addition, the local image processing unit 113 (FIG. 3) of the first robot 100(1) generates a first local image Is(1) by taking a picture using the camera 152 (S108), and transmits the generated first local image Is(1) to the external network NET via the communication interface 130 (S110). In the example of FIG. 7, since two first users U(1) face the first robot 100(1), the first local image Is(1) includes images representing the two first users U(1) (more specifically, actual images of the first users U(1)). Note that this first local image Is(1) corresponds to a "remote image" when viewed from the second robot 100(2).

[0068] Similarly, the robot operation control unit 111 (FIG. 3) of the second robot 100(2) connects to the external network NET via the communication interface 130 (S102). In addition, the local image processing unit 113 (FIG. 3) of the second robot 100(2) generates a second local image Is(2) by taking a picture using the camera 152 (S108), and transmits the generated second local image Is(2) to the external network NET via the communication interface 130 (S110). In the example of FIG. 7, since two second users U(2) face the second robot 100(2), the second local image Is(2) includes images representing the two second users U(2) (more specifically, actual images of the second users U(2)). Note that this second local image Is(2) corresponds to a "remote image" when viewed from the first robot 100(1).

[0069] The remote image processing unit 112 of the first robot 100(1) acquires the second local image Is(2) transmitted from the second robot 100(2) via the communication interface 130, and displays the acquired second local image Is(2) as a remote image Ir on the display 151 (S111). By displaying the second local image Is(2) as the remote image Ir on the display 151, each first user U(1) at the first point P(1) can visually recognize the actual image of each second user U(2) displayed on the display 151 of the first robot 100(1). In other words, each first user U(1) at the first point P(1) can recognize that multiple second users U(2) have taken over the first robot 100(1).

[0070] Similarly, the remote image processing unit 112 of the second robot 100(2) acquires the first local image Is(1) transmitted from the first robot 100(1) via the communication interface 130, and displays the acquired first local image Is(1) as a remote image Ir on the display 151 (S111). By displaying the first local image Is(1) as the remote image Ir on the display 151, each second user U(2) at the second point P(2) can visually recognize the actual image of each first user U(1) displayed on the display 151 of the second robot 100(2). In other words, each second user U(2) at the second point P(2) can recognize that multiple first users U(1) have taken over the second robot 100(2).

[0071] In this state, two first users U(1) are actually located at the first point P(1), and two second users U(2) who are actually located at the second point P(2) are virtually located at the first point P(1) by transferring to the first robot 100(1). Similarly, two second users U(2) are actually located at the second point P(2), and two first users U(1) who are actually located at the first point P(1) are virtually located at the second point P(2) by transferring to the second robot 100(2). Therefore, four users U who are actually located at different points can virtually gather at the first point P(1) or the second point P(2) and communicate with each other through vision and hearing.

[0072] As in the first embodiment, the robot arm control unit 117 (FIG. 3) of each robot 100 monitors whether or not there is a robot arm operation instruction (S112), and when there is a robot arm operation instruction (S112: YES), operates the robot arm 155 according to the instruction (S114). Also, the movement control unit 116 (FIG. 3) of each robot 100 monitors whether or not a movement condition is satisfied (S116), and when the movement condition is satisfied (S116: YES), operates the movement mechanism 156 to move the robot 100 (S118).

[0073] Unless an end instruction (S120) is given in each device, the above process is repeated, and communication between users U located at different locations is continuously performed. When an end instruction is given in each device (S120: YES), the remote communication process by the remote communication system 10a ends.

[0074] As described above, in each robot 100 constituting the remote communication system 10a of the second embodiment, similarly to the first embodiment, the movement control unit 116 controls the movement mechanism 156 to move the robot 100, the remote image processing unit 112 displays the remote image Ir including an image showing the users U at multiple remote locations on the display 151, and the local image processing unit 113 makes the users U at multiple remote locations visually recognize the local image Is, which is an image captured by the camera 152. Therefore, each robot 100 of the second embodiment functions as a collective telexistence device that can transfer the users U at multiple remote locations to the robot 100. Therefore, each robot 100 of the second embodiment can realize real-time communication between the users U at multiple remote locations who have transferred to the robot 100 and the user U who is actually located at the current location of the robot 100.

[0075] C. Variations: The technology disclosed in this specification is not limited to the above-described embodiments, and can be modified in various forms without departing from the spirit of the invention. For example, the following modifications are also possible.

[0076] The configuration of the remote communication system 10 in the above embodiment is merely an example, and various modifications are possible. For example, in the above first embodiment, the HMD 200 is used as a device for the remote user Ur to access the information space VS, but a device other than the HMD 200 (e.g., a PC, a smartphone, a tablet terminal, smart glasses, etc.) may be used. In addition, in the above second embodiment, the robot 100 is used as a device for communication at both of the two points P, but a device other than the robot 100 (e.g., a PC, a smartphone, a tablet terminal, smart glasses, etc.) may be used at either of the two points P.

[0077] The configuration of the robot 100 in the above embodiment is merely an example, and various modifications are possible. For example, in the above embodiment, the display 151 is a 360-degree display, but the display 151 does not necessarily have to be a 360-degree display. Also, in the above embodiment, the camera 152 is a 360-degree camera, but the camera 152 does not necessarily have to be a 360-degree camera. Also, in the above embodiment, the microphone 153 is a 360-degree microphone, but the microphone 153 does not necessarily have to be a 360-degree microphone. Also, in the above embodiment, the speaker 154 is a directional speaker, but the speaker 154 does not necessarily have to be a directional speaker.

[0078] In the above embodiment, the robot 100 may not include at least one of the microphone 153, the speaker 154, and the robot arm 155.

[0079] The contents of the remote communication process in the above embodiment are merely examples and can be modified in various ways.

[0080] In the above-described embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. [Explanation of symbols]

[0081] 10: Remote communication system 100: Robot 110: Control unit 111: Robot operation control unit 112: Remote image processing unit 113: Local image processing unit 114: Remote audio processing unit 115: Local audio processing unit 116: Movement control unit 117: Robot arm control unit 118: Information space construction unit 120: Memory unit 130: Communication interface 140: Operation input unit 151: Display 152: Camera 153: Microphone 154: Speaker 155: Robot arm 156: Movement mechanism 157: Wheel 190: Bus 200: HMD 210: Control unit 220: Memory unit 230: Communication interface 240: Operation input unit 251: Right eye display execution unit 252: Left eye display execution unit 253: Microphone 254: Speaker 255: Head movement detection unit 290: Bus CP: Robot control program Ir: Remote image Is: Local image NET: External network Pr1: First remote location Pr2: Second remote location Pr3: Third remote location Ps: Local Ur1: First remote user Ur2: Second remote user Ur3: Third remote user Ura: Avatar Us: Local user VS: Information space

Claims

1. A robot, A display and A camera and A communication interface; A movement mechanism for moving the robot; A control unit; Equipped with The control unit is a movement control unit that controls the movement mechanism to move the robot; a remote image processing unit that acquires remote images including images of a plurality of remote users located at locations different from a current location of the robot from an external network via the communication interface and displays the acquired remote images on the display; a local image processing unit that transmits local images captured by the camera to the external network via the communication interface so that the local images can be viewed by the multiple remote users; having The control unit further includes an information space construction unit that constructs an information space shared by the plurality of remote users on the external network, The on-site image processing unit projects the on-site image into the information space, thereby allowing the on-site image to be viewed by the multiple remote users virtually located in the information space.

2. The robot according to claim 1 , The remote image includes images of avatars of the multiple remote users.

3. A robot, A display and A camera and A communication interface; A movement mechanism for moving the robot; A control unit; Equipped with The control unit is a movement control unit that controls the movement mechanism to move the robot; a remote image processing unit that acquires remote images including images of a plurality of remote users located at locations different from a current location of the robot from an external network via the communication interface and displays the acquired remote images on the display; a local image processing unit that transmits local images captured by the camera to the external network via the communication interface so that the local images can be viewed by the multiple remote users; having The robot, wherein the display is a 360 degree display.

4. A robot, A display and A camera and A communication interface; A movement mechanism for moving the robot; A control unit; Equipped with The control unit is a movement control unit that controls the movement mechanism to move the robot; a remote image processing unit that acquires remote images including images of a plurality of remote users located at locations different from a current location of the robot from an external network via the communication interface and displays the acquired remote images on the display; a local image processing unit that transmits local images captured by the camera to the external network via the communication interface so that the local images can be viewed by the multiple remote users; having The robot, wherein the camera is a 360 degree camera.

5. A robot, A display and A camera and A communication interface; A movement mechanism for moving the robot; A control unit; Equipped with The control unit is a movement control unit that controls the movement mechanism to move the robot; a remote image processing unit that acquires remote images including images of a plurality of remote users located at locations different from a current location of the robot from an external network via the communication interface and displays the acquired remote images on the display; a local image processing unit that transmits local images captured by the camera to the external network via the communication interface so that the local images can be viewed by the multiple remote users; having In addition, it is equipped with a 360-degree microphone, The control unit further has a local voice processing unit that transmits local voice, which is voice acquired by the 360-degree microphone, to the external network via the communication interface so that the multiple remote users can hear the local voice and recognize the direction of the sound source.

6. A robot, A display and A camera and A communication interface; A movement mechanism for moving the robot; A control unit; Equipped with The control unit is a movement control unit that controls the movement mechanism to move the robot; a remote image processing unit that acquires remote images including images of a plurality of remote users located at locations different from a current location of the robot from an external network via the communication interface and displays the acquired remote images on the display; a local image processing unit that transmits local images captured by the camera to the external network via the communication interface so that the local images can be viewed by the multiple remote users; having In addition, it is equipped with a directional speaker, The control unit further has a remote voice processing unit that acquires remote voice, which is voice emitted from the multiple remote users, from the external network via the communication interface, and outputs the acquired remote voice from the directional speaker so as to enable the direction in which each of the remote users is located to be recognized.

7. A robot, A display and A camera and A communication interface; A movement mechanism for moving the robot; A control unit; Equipped with The control unit is a movement control unit that controls the movement mechanism to move the robot; a remote image processing unit that acquires remote images including images of a plurality of remote users located at locations different from a current location of the robot from an external network via the communication interface and displays the acquired remote images on the display; a local image processing unit that transmits local images captured by the camera to the external network via the communication interface so that the local images can be viewed by the multiple remote users; having In addition, it is equipped with a robot arm, The control unit further has a robot arm control unit that acquires operation instructions from the multiple remote users from the external network via the communication interface and operates the robot arm in accordance with the acquired operation instructions.

8. A robot control method for controlling a robot including a display, a camera, a communication interface, and a moving mechanism, comprising: controlling the movement mechanism to move the robot; acquiring remote video including video of a plurality of remote users located at locations different from a current location of the robot from an external network via the communication interface, and displaying the acquired remote video on the display; transmitting a local image captured by the camera to the external network via the communication interface so that the local image can be viewed by the multiple remote users; constructing an information space shared by the plurality of remote users on the external network; Equipped with A robot control method, wherein the step of transmitting the on-site image is a step of projecting the on-site image into the information space so that the on-site image can be viewed by the multiple remote users virtually located in the information space.

9. A computer program for controlling a robot including a display, a camera, a communication interface, and a moving mechanism, comprising: On the computer, A process of controlling the movement mechanism to move the robot; acquiring remote video including video of a plurality of remote users located at locations different from a current location of the robot from an external network via the communication interface, and displaying the acquired remote video on the display; a process of transmitting a local image captured by the camera to the external network via the communication interface so that the local image can be viewed by the multiple remote users; A process of constructing an information space shared by the plurality of remote users on the external network; Run the command, A computer program, wherein the process of transmitting the on-site image is a process of projecting the on-site image into the information space, thereby making the on-site image visible to the multiple remote users virtually located in the information space.

Images