Remote robot system and control method for remote robot system

Abstract

Provided are a remote robot system and a control method for the remote robot system, which can appropriately respond to users. The remote robot system includes: a robot that performs a predetermined action including collection of local information around; a guide terminal that can remotely operate the action of the robot; and a participant terminal that can remotely communicate with the guide terminal. The participant terminal includes: a display panel that outputs the local information obtained from the robot to the user; and an interest detection unit that detects the user's interest in the output local information.

Description

Technical Field

[0001] The present invention relates to a remote robot system and a control method for a remote robot system. Background Art

[0002] In recent years, users have been able to remotely operate an autonomously traveling robot from an operation terminal. In addition, by using a telepresence robot equipped with a camera, a microphone, a monitor, etc., which is an example of such a robot, users can also remotely communicate and have a virtual experience. Moreover, with the spread of remote services such as Web conferencing services, the widespread use of robots that can perform remote operations and remote communication is expected.

[0003] In relation to user operations, for example, Japanese Unexamined Patent Application Publication No. 2016-224583 discloses the following technique: in an analysis server device, a heat map is used to analyze the browsing behavior of a Web page performed by a user. In addition, Japanese Unexamined Patent Application Publication No. 2014-50465 discloses the following technique: in an image processing device that displays a medical image such as an X-ray CT image, a region of interest of a user is specified for diagnosis. Summary of the Invention

[0004] By using a robot such as a telepresence robot, a user can remotely participate in and virtually experience a remote sightseeing tour or the like. For example, in a remote robot system for a remote sightseeing tour or the like, a tour guide (operator) remotely operates the robot, and a user (participant) who participates in the tour communicates with the guide remotely. However, in such a remote robot system, since the guide and the user are remote, it is difficult to appropriately respond to the user. In addition, the technique disclosed in Japanese Unexamined Patent Application Publication No. 2016-224583 is a technique for analyzing the browsing of a Web page, and the technique disclosed in Japanese Unexamined Patent Application Publication No. 2014-50465 is a technique for specifying a diagnosis region of a medical image, and these techniques cannot solve such a problem.

[0005] The present invention has been completed to solve such a problem, and provides a remote robot system and a control method for a remote robot system that can appropriately respond to a user.

[0006] A remote robot system according to one aspect of the present invention includes: a robot that performs a predetermined operation including collection of local information around it; a first remote terminal capable of remotely operating the operation of the robot; and a second remote terminal capable of remotely communicating with the first remote terminal. The second remote terminal includes: an information output unit that outputs the local information obtained from the robot to a user; and an interest detection unit that detects the user's interest in the output local information. Thus, since the interest of the user of the second remote terminal can be detected, even when the first remote terminal and the second remote terminal are far apart, the interest of the user can be grasped and the user can be appropriately responded to.

[0007] In addition, the interest detection unit may also detect the operation state of the user with respect to the output local information, and detect the user's interest based on the detected operation state. Moreover, the operation state may also include the operation position, the number of operations, or the operation time during the user's operation on the display screen on which the local information is displayed. Thus, the interest can be detected with high accuracy according to the state of the user's operation.

[0008] In addition, the interest detection unit may also detect the vocalization state of the user with respect to the output local information, and detect the user's interest based on the detected vocalization state. Moreover, the vocalization state may also include the sound pressure of the sound emitted by the user, the vocalization time, or the recognition result of the sound. Thus, the interest can be detected with high accuracy according to the state of the sound emitted by the user.

[0009] In addition, the interest detection unit may also detect the gaze state of the user with respect to the output local information, and detect the user's interest based on the detected gaze state. Moreover, the gaze state may also include the orientation of the user's face, the orientation of the user's line of sight, the gaze position, or the gaze time. Thus, the interest can be detected with high accuracy according to the state of the user's gaze.

[0010] In addition, the second remote terminal may also include a notification unit that notifies the first remote terminal of interest information indicating the detected interest. Thus, the interest of the user detected by the second remote terminal can be reliably fed back to the first remote terminal.

[0011] In addition, the first remote terminal may also be provided with an interest attention output unit that outputs the notified interest attention information. Moreover, the interest attention output unit may also highlight and display the position corresponding to the interest attention information on the display screen where the local information is displayed. In addition, the interest attention output unit may also display the temporal change of the interest attention information on the display screen where the local information is displayed. Thereby, in the first remote terminal, it is possible to grasp at a glance the interest attention of the user of the second remote terminal.

[0012] In addition, the first remote terminal may also be provided with a remote control determination unit that determines the remote operation method of the robot based on the notified interest attention information. Moreover, the remote control determination unit may also determine the movement method of the robot including the movement direction, movement path, or movement speed of the robot based on the interest attention information. In addition, the remote control determination unit may also determine the information acquisition method of the robot including the imaging direction of the image of the robot or the sound collection direction based on the interest attention information. Thereby, it is possible to appropriately remotely control the robot according to the interest attention of the user of the second remote terminal.

[0013] In a control method of a remote robot system according to an aspect of the present invention, the remote robot system includes: a robot that performs a predetermined action including collection of local information around; a first remote terminal that can remotely operate the action of the robot; and a second remote terminal that can communicate remotely with the first remote terminal. The second remote terminal outputs the local information obtained from the robot to the user, and the second remote terminal detects the user's interest attention to the output local information. Thereby, since it is possible to detect the interest attention of the user of the second remote terminal, even when the first remote terminal and the second remote terminal are far apart, it is possible to grasp the user's interest attention and appropriately respond to the user.

[0014] According to the present invention, it is possible to provide a remote robot system and a control method of a remote robot system that can appropriately respond to the user.

[0015] The above and other objects, features, and advantages of the present disclosure will be more fully understood from the following detailed description and the accompanying drawings given by way of illustration only, and should not be considered as a limitation of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural diagram showing a structural example of a remote robot system according to Embodiment 1.

[0017] Figure 2It is an external perspective view showing an example of the external structure of the robot according to Embodiment 1.

[0018] Figure 3 It is a block diagram showing an example of the functional structure of the robot according to Embodiment 1.

[0019] Figure 4 It is a block diagram showing an example of the functional structure of the guidance terminal according to Embodiment 1.

[0020] Figure 5 It is an image diagram showing an example of the structure of the display screen of the guidance terminal according to Embodiment 1.

[0021] Figure 6 It is a block diagram showing an example of the functional structure of the participant terminal according to Embodiment 1.

[0022] Figure 7 It is an image diagram showing an example of the structure of the display screen of the participant terminal according to Embodiment 1.

[0023] Figure 8 It is a timing chart showing an example of the operation of the remote robot system according to Embodiment 1.

[0024] Figure 9 It is an image diagram showing an example of the screen display of the guidance terminal according to Embodiment 1.

[0025] Figure 10 It is an image diagram showing an example of the screen display of the guidance terminal according to Embodiment 1.

[0026] Figure 11A It is an image diagram showing an example of the screen display of the guidance terminal according to Embodiment 1.

[0027] Figure 11B It is an image diagram showing an example of the screen display of the guidance terminal according to Embodiment 1.

[0028] Figure 12 It is an image diagram showing an example of the screen display of the guidance terminal according to Embodiment 1.

[0029] Figure 13 It is a block diagram showing an example of the functional structure of the guidance terminal according to Embodiment 2.

[0030] Figure 14 It is a timing chart showing an example of the operation of the remote robot system according to Embodiment 2. Detailed implementation manners

[0031] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the respective drawings, the same reference numerals are assigned to the same elements, and repeated descriptions are omitted as necessary. In addition, the present invention is described by way of embodiments, but the invention according to the claims is not limited to the following embodiments. Further, all the structures described in the embodiments are not necessarily essential as means for solving the problems.

[0032] (Embodiment 1)

[0033] First, Embodiment 1 of the present invention will be described. Figure 1 A structural example of the remote robot system 10 according to this embodiment is shown. As Figure 1 shown, the remote robot system 10 according to this embodiment includes a robot 100, a guide terminal (first remote terminal) 200, a plurality of participant terminals (second remote terminals) 300 (for example, 300-1 to 300-3), and a system server 400.

[0034] The remote robot system 10 is a system in which the robot 100 at a sightseeing spot (first environment) is remotely operated by operating the guide terminal 200 by a guide (tour guide) G, and the participant P can virtually experience a sightseeing tour via the remotely operated robot 100. In the remote robot system 10, a plurality of participants P (for example, P1 to P3) and the guide G can be connected to the robot 100, which is, for example, a remote presence robot, via a terminal device, and while sharing the images / sounds acquired by the robot 100, enjoy a remote sightseeing tour via the robot 100. One of the users connected to the robot 100 is the guide G. The guide G remotely controls the robot 100 to change the traveling direction and perspective of the robot 100 while explaining the sightseeing spot. The participant P enjoys the sightseeing from a remote area while viewing and listening to the images / sounds of the robot 100 and the guide G. At this time, the guide G needs to conduct the guide according to the interest concerns of the participant P, but it is difficult to observe the reactions of the participant P and the like on the connection terminal of the guide G, making it difficult to conduct an appropriate guide. Therefore, in this embodiment, by detecting the interest concerns of the participant P and then feeding back the detected interest concerns to the guide G, the guide G can conduct the guidance of the tour according to the interest concerns of the participant P.

[0035] In addition, each participant can also perform remote operation and virtual experience by providing a plurality of robots 100. Further, by providing a plurality of guide terminals 200, a plurality of guides G can remotely operate a plurality of robots 100.

[0036] As Figure 1As shown, the robot 100, the guide terminal 200, and multiple participant terminals 300 are respectively arranged in remote environments (the first environment, the second environment, and the third environment), and are connected to the system server 400 via a wireless router 600 and the Internet 500. For example, the system server 400 provides data, processing, services, etc. required for remote operation and remote communication for the robot 100, the guide terminal 200, and the participant terminals 300. For example, communication can be performed between the wireless router 600 and the robot 100, the guide terminal 200, and the participant terminals 300 via a wireless LAN, but communication can also be performed via mobile phone communication such as LTE (Long Term Evolution) or 5G (5th Generation Mobile Communication System). By using mobile phone communication in the communication of the robot 100, the usage location of the robot 100 is extended, so the robot 100 in various locations can be remotely operated. In this case, the wireless router 600 can also be a base station for mobile phone communication.

[0037] The robot 100 can perform predetermined actions in the first environment, including collecting local information (including images, sounds, etc.) around it. By operating the guide terminal 200 (operation terminal) by the remote operator, i.e., the guide G, in the second environment far from the first environment, the robot 100 is remotely operated via the system server 400 connected to the Internet 500. The robot 100 receives various operation signals from the guide terminal 200 and performs movement actions, camera actions, etc. In addition, the guide G and the participant P can also perform remote communication via the robot 100.

[0038] The robot 100 uses the camera 131 (imaging unit) to capture images of the space around the robot 100, and sends the captured imaging images (videos) to the guide terminal 200 and the participant terminals 300 via the Internet 500. In addition, the microphone 132 is used to pick up the sounds around the robot 100, and the picked-up sounds are sent to the guide terminal 200 and the participant terminals 300 via the Internet 500. Moreover, sounds are received from the guide terminal 200 and the participant terminals 300 via the Internet 500, and the received sounds are output from the speaker 133. The robot 100 can also perform gripping actions and the like implemented by the hand 124 according to the operations of the guide terminal 200. Thus, various virtual experiences can be realized during sightseeing tours.

[0039] In addition, in the present embodiment, the Internet 500 is used as the connected network, but other networks such as an intranet can also be used. Additionally, instead of via a network, the robot 100 can be directly connected to the guide terminal 200 and the participant terminal 300 using short-range communication such as Bluetooth (registered trademark in Japan, Bluetooth). That is, remote control and remote communication can be performed without passing through the Internet 500 and the system server 400.

[0040] The guide terminal 200 is a terminal operated by the guide G in the second environment, remotely operates the robot 100, and conducts remote communication with the participant terminal 300 and the robot 100. The participant terminal 300 (e.g., 300-1 to 300-3) is a terminal operated by the participant P (e.g., participant P1 to P3) in the third environment (e.g., the third-1 environment to the third-3 environment), and conducts remote communication with the guide terminal 200, other participant terminals 300, and the robot 100.

[0041] The guide terminal 200 and the participant terminal 300 are, for example, tablet terminals, and have display panels 201 and 301 on which a touch panel is overlapped. The display panels 201 and 301 can display the captured images received from the robot 100, and the guide G and the participant P can indirectly visually recognize the appearance of the space area around the robot 100. In other words, the display panel 201 is an information output unit that outputs the local information of the robot 100 to the guide G, and the display panel 301 is an information output unit that outputs the local information of the robot 100 to the participant P (user).

[0042] In addition, the guide terminal 200 and the participant terminal 300 can output the sound received from the robot 100, and the guide G and the participant P can indirectly hear the sound around the robot 100. The guide G operates the guide terminal 200 via the touch panel, thereby being able to generate various operation signals for causing the robot 100 to move. The generated operation signals are sent to the robot 100 via the Internet 500.

[0043] In addition, the participant terminal 300 includes an interest detection unit 321 that detects the participant P's interest in the local information of the output robot 100. The interest detection unit 321 generates interest information representing the participant P's interest in response to the participant P operating on the display screen of the display panel 301 via the touch panel or the like. The generated interest information is sent to the guide terminal 200 via the Internet 500. In addition, the guide terminal 200 includes an interest analysis unit 221 that analyzes the interest information received from the participant terminal 300. The analyzed interest of the participant P is displayed on the display panel 201. The display panel 201 is also an interest output unit that outputs the interest of the participant P.

[0044] Figure 2 FIG. is an external perspective view showing an example of the external structure of the robot 100 according to the present embodiment. As Figure 2 shown, in the robot 100 according to the present embodiment, a base portion 110 and a main body portion 120 are provided as main components.

[0045] The base portion 110 supports two drive wheels 111 and one caster 112 that are in contact with the traveling surface inside a cylindrical frame. The two drive wheels 111 are configured such that their rotation axes are aligned with each other. Each drive wheel 111 is independently rotationally driven by a motor (not shown). The caster 112 is a driven wheel, and the caster 112 is provided in such a manner that a rotation axis extending vertically from the base portion 110 is away from the rotation axis of the wheel to axially support the wheel, and follows in a manner imitating the moving direction of the base portion 110.

[0046] In addition, the base portion 110 is provided with a laser scanner 134 at the peripheral portion of the upper surface. The laser scanner 134 scans a certain range in the horizontal plane for each step angle and outputs whether there are obstacles in each direction. Moreover, the laser scanner 134 outputs the distance to the obstacle when there is an obstacle.

[0047] The main body portion 120 mainly includes a trunk portion 121 mounted on the upper surface of the base portion 110, a head portion 122 placed on the upper surface of the trunk portion 121, an arm 123 supported on the side surface of the trunk portion 121, and a hand portion 124 provided at the front end of the arm 123. The arm 123 and the hand portion 124 are driven by a motor (not shown) to clamp an object to be clamped. The trunk portion 121 can rotate relative to the base portion 110 about a vertical axis by the driving force of a motor (not shown).

[0048] The head 122 mainly includes a camera 131, a microphone 132, a speaker 133, and a display panel 141. The camera 131 is disposed, for example, on the front surface of the head 122 such that the front direction of the head 122 is the imaging direction. The camera 131 can be either a two-dimensional camera or a stereo camera. For example, in the case of a stereo camera, it has a structure in which two camera units having the same field of view angle are arranged at intervals from each other, and outputs imaging signals obtained by imaging with each camera unit.

[0049] The microphone 132 is disposed, for example, on the front surface of the head 122 such that the front direction of the head 122 is the sound collection direction. The microphone 132 can be either a single-directional microphone or an omnidirectional microphone. The speaker 133 is disposed, for example, on the front surface or side surface of the head 122 such that the front direction of the head 122 is the output direction. The speaker 133 can be either a monaural speaker or a stereo speaker.

[0050] The display panel 141 is disposed on the front surface of the head 122 such that the front direction of the head is the display direction. The display panel 141 is, for example, a liquid crystal panel, and displays the face of the set character using animation, or displays information related to the robot 100 using text or icons. If the face of the character is displayed on the display panel 141, it can give people around the impression that the display panel 141 is a virtual face. The display panel 141 can also display the images of the guide G and the participant P received from the guide terminal 200 and the participant terminal 300.

[0051] The head 122 can rotate about a vertical axis relative to the torso 121 by the driving force of a motor (not shown). Therefore, the camera 131 can image in any direction, the microphone 132 can collect sounds in any direction, the speaker 133 can output sounds in any direction, and in addition, the display panel 141 can present the display content in any direction.

[0052] Figure 3 It is a block diagram showing a functional structure example of the robot 100 according to the present embodiment. As Figure 3 shown, in the robot 100, as functional modules, in addition to the above-mentioned display panel 141, camera 131, microphone 132, speaker 133, and laser scanner 134, it further includes a control unit 150, a base driving unit 142, an upper body driving unit 143, a memory 160, a communication unit 170, and a GPS unit 135.

[0053] In addition, the main elements related to the remote operation and remote communication implemented using the wizard terminal 200 and the participant terminal 300 are described herein. However, other elements may also be provided as the structure of the robot 100 as needed, and other elements that contribute to the remote operation and remote communication may be added.

[0054] The control unit 150 is, for example, a CPU (Central Processing Unit), and executes the overall control of the robot 100 and various arithmetic processes by executing a control program read out from the memory 160. In addition, the control unit 150 also serves as a functional execution unit that executes various operations and controls related to control.

[0055] The control unit 150 is housed in, for example, a control unit provided in the trunk 121. The base drive unit 142 includes drive wheels 111 and a drive circuit and a motor for driving the drive wheels 111. The control unit 150 executes the rotation control of the drive wheels by sending a drive signal to the base drive unit 142. In addition, the control unit 150 receives feedback signals such as an encoder from the base drive unit 142 to grasp the moving direction and moving speed of the base unit 110.

[0056] The upper body drive unit 143 includes the arm 123 and the hand 124, the trunk 121 and the head 122, and a drive circuit and a motor for driving them. The control unit 150 realizes the gripping action and gestures by sending a drive signal to the upper body drive unit 143. In addition, the control unit 150 receives feedback signals such as an encoder from the upper body drive unit 143 to grasp the positions and moving speeds of the arm 123 and the hand 124, and the orientations and rotational speeds of the trunk 121 and the head 122.

[0057] The display panel 141 receives and displays the image signal generated by the control unit 150. The control unit 150 generates an image signal of a character or the like based on the information in the memory 160, the received information, etc., and displays it on the display panel 141. For example, when the display panel 141 has a touch panel, the display panel 141 outputs a touch signal to the control unit 150.

[0058] The camera 131 performs imaging of the front of the robot 100, for example, in accordance with a request from the control unit 150, and outputs the imaging signal generated by the imaging to the control unit 150. The control unit 150 uses the imaging signal from the camera 131 to execute image processing or transforms the imaging signal into an imaging image in accordance with a predetermined format.

[0059] The microphone 132 picks up the sounds around the robot 100 in accordance with a request from the control unit 150, and outputs the sound signal generated by the sound pickup to the control unit 150. The control unit 150 uses the sound signal from the microphone 132 to perform sound processing or transforms the imaging signal in accordance with a predetermined format.

[0060] The speaker 133 outputs sounds sent from, for example, the guide terminal 200 and the participant terminal 300. The control unit 150 outputs the sound signals received from the guide terminal 200 and the participant terminal 300 to the speaker 133 via the communication unit 170, and the speaker 133 outputs the sound signals from the control unit 150 to the surroundings.

[0061] The laser scanner 134 detects whether there is an obstacle in the moving direction of the robot 100 in accordance with a request from the control unit 150, and outputs a detection signal as its detection result to the control unit 150. The GPS unit 135 detects the position of the robot 100 based on the GPS signals received from GPS (Global Positioning System) satellites in accordance with a request from the control unit 150, and outputs a position signal (position information) as its detection result to the control unit 150. In addition, the position of the robot 100 may be detected by other methods, not limited to GPS.

[0062] The memory 160 is a non-volatile storage medium, for example, a solid state drive is used. In addition to storing the control program for controlling the robot 100, the memory 160 also stores various parameter values, functions, look-up tables, etc. used in control and calculation. For example, the memory 160 stores a map DB 161, which is a database describing the map information of the space of the first environment where autonomous movement is scheduled. In the map DB 161, the map information to be the object is obtained from, for example, the system server 400 and stored in accordance with the action range of the robot 100.

[0063] The communication unit 170 is, for example, a wireless LAN unit that performs wireless communication with the wireless router 600. The communication unit 170 transmits and receives control signals, data, etc. to and from the guide terminal 200 and the participant terminal 300 via the wireless router 600. For example, the communication unit 170 receives a specified signal and an operation instruction sent from the guide terminal 200 and outputs them to the control unit 150. The specified signal is a signal that requests a captured image and a sound signal generated by the guide terminal 200 by the user specifying a spatial area to be observed. In addition, according to the control of the control unit 150, the communication unit 170 transmits the captured image obtained by the camera 131 and the sound obtained by the microphone 132 to the guide terminal 200 and the participant terminal 300. The communication unit 170 may also transmit the position information detected by the GPS unit 135 to the guide terminal 200 and the participant terminal 300, and transmit the map information stored in the map DB 161 to the guide terminal 200 and the participant terminal 300. In addition, the communication unit 170 receives images and sounds sent from the guide terminal 200 and the participant terminal 300 and outputs them to the control unit 150.

[0064] Figure 4 It is a block diagram showing a functional structure example of the guide terminal 200 according to the present embodiment. As Figure 4 shown, the guide terminal 200 includes an arithmetic unit 220, a display panel 201, an input unit 202, a microphone 203, a speaker 204, a camera 205, a memory 230, and a communication unit 240 as functional modules.

[0065] In addition, the main elements related to the remote operation of the robot 100 or the remote communication with the robot 100 and the participant terminal 300 are described here, but other elements may also be provided as the structure of the guide terminal 200 as needed, and other elements that contribute to the processing for remote operation and remote communication may be added.

[0066] The arithmetic unit (control unit) 220 is, for example, a CPU, and executes the overall control and various arithmetic processes of the guide terminal 200 by executing a control program read from the memory 230. The display panel 201 is, for example, a liquid crystal panel, and displays, for example, a captured image sent from the robot 100, the participant terminal 300, the map information, and a captured image captured by the camera 205.

[0067] The input unit 202 includes a touch panel overlapped on the display panel 201, push buttons provided at the peripheral portion of the display panel 201, etc. The input unit 202 accepts the operation of the guide G to generate an operation signal and outputs it to the arithmetic unit 220.

[0068] The microphone 203 picks up the sounds around the guidance terminal 200 according to the requirements from the operation unit 220, and outputs the sound signals generated by the sound pickup to the operation unit 220. The speaker 204 outputs the sounds sent from, for example, the robot 100 and the participant terminal 300. The camera 205 picks up the image of the guide G in front of, for example, the guidance terminal 200 (in front of the display panel 201) according to the requirements from the operation unit 220, and outputs the captured image signals generated by the imaging to the operation unit 220.

[0069] The memory 230 is a non-volatile storage medium, for example, a solid state drive is used. In addition to storing the control program for controlling the guidance terminal 200, the memory 230 also stores various parameter values, functions, lookup tables, etc. used in control and operation.

[0070] The communication unit 240 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 600. The communication unit 240 transmits and receives control signals, data, etc. to and from the robot 100 and the participant terminal 300 via the wireless router 600. For example, the communication unit 240 receives the captured images, sound signals, map information, etc. sent from the robot 100 and the participant terminal 300, and outputs them to the operation unit 220. In addition, the communication unit 240 collaborates with the operation unit 220 to send the specified signals, operation signals, captured images, sound signals, etc. to the robot 100 and the participant terminal 300.

[0071] The operation unit 220 also undertakes the function of a functional operation unit that executes various processes and operations. For example, the operation unit includes an interest analysis unit 221 and a display control unit 222. The interest analysis unit 221 analyzes the interest of the participant P according to the interest information sent from the participant terminal 300. The display control unit 222 displays the analysis results of the interest of the participant P and the like on the display panel 201. For example, information indicating the interest of the participant P is overlapped and displayed on the captured image or the like displayed on the display panel 201. It can also be said that the output unit for outputting the interest is formed by the cooperation of the display control unit 222 and the display panel 201.

[0072] Figure 5 The structural example of the guidance screen 210 displayed on the display panel 201 of the guidance terminal 200 according to the present embodiment is shown. As Figure 5 shown, for example, the guidance screen 210 includes a robot image display area 211, a map display area 212, a guidance image display area 213, a participant image display area 214, an operation panel display area 215, and an interest display area 216. In addition, Figure 5 The display of each area in is an example and is not limited to these.

[0073] The robot image display area 211 displays a captured image around the robot 100 captured and transmitted by the robot 100. In addition, the robot image display area 211 overlays and displays, for example, a heat map corresponding to the interest of the participant P on the captured image of the robot 100.

[0074] The map display area 212 displays map information and location information around the robot 100 transmitted by the robot 100. The map display area 212 displays, for example, information such as tourist attractions around the robot 100, the current location of the robot 100, the movement path, the destination, the orientation, etc.

[0075] The guide image display area 213 displays a captured image (guide image) of the guide G (operator) captured by the camera 205, mainly including the face. The participant image display area 214 displays captured images (participant images) of a plurality of participants P captured and transmitted by a plurality of participant terminals 300, mainly including the face.

[0076] The operation panel display area 215 displays an operation panel for operating (remotely operating) the robot 100. In the operation panel, there are buttons and the like for operating the moving direction, moving speed, camera orientation, etc. of the robot 100.

[0077] The interest display area 216 displays information related to the interest of the participant P. The interest display area 216 displays, for example, a rating table indicating the level of interest and a graph indicating the temporal change of the interest.

[0078] Figure 6 It is a block diagram showing an example of the block structure of the participant terminal 300 according to the present embodiment. As Figure 6 shown, the participant terminal 300 includes, as functional modules, an arithmetic unit 320, a display panel 301, an input unit 302, a microphone 303, a speaker 304, a camera 305, a memory 330, and a communication unit 340.

[0079] In addition, the main elements related to remote communication between the robot 100, the guide terminal 200, and other participant terminals 300 are described here, but other elements may also be included as the structure of the participant terminal 300, and other elements contributing to the processing for remote communication may be added.

[0080] The operation unit (control unit) 320 is, for example, a CPU, and executes the overall control of the participant terminal 300 and various arithmetic processes by executing a control program read from the memory 330. The display panel 301 is, for example, a liquid crystal panel, and displays, for example, captured images sent from the robot 100, the guidance terminal 200, other participant terminals 300, map information, captured images obtained by capturing images with the camera 305, and the like.

[0081] The input unit 302 includes a touch panel disposed overlapping the display panel 301, push buttons provided at the peripheral portion of the display panel 301, and the like. The input unit 302 accepts the operation of the participant P to generate an operation signal and outputs it to the operation unit 320.

[0082] The microphone 303 picks up the sounds around the participant terminal 300 in accordance with a request from the operation unit 320, and outputs the sound signal generated by the sound pickup to the operation unit 320. The speaker 304 outputs, for example, sounds sent from the robot 100, the guidance terminal 200, other participant terminals 300. The camera 305 captures an image of the participant P in front of the participant terminal 300 (in front of the display panel 301) in accordance with a request from the operation unit 320, and outputs the captured signal generated by the image capture to the operation unit 320.

[0083] The memory 330 is a non-volatile storage medium, and for example, a solid state drive is used. In addition to storing a control program for controlling the participant terminal 300, the memory 330 also stores various parameter values, functions, look-up tables, etc. used in control and arithmetic operations.

[0084] The communication unit 340 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 600. The communication unit 340 transmits and receives control signals, data, etc. to and from the robot 100, the guidance terminal 200, other participant terminals 300 via the wireless router 600. For example, the communication unit 340 receives captured images, sound signals, map information sent from the robot 100, the guidance terminal 200, other participant terminals 300, and outputs them to the operation unit 320. In addition, the communication unit 340 cooperates with the operation unit 320 to send captured images, sound signals, etc. to the guidance terminal 200, other participant terminals 300.

[0085] The operation unit 320 also undertakes the function of a functional operation unit that executes various processes and operations. For example, the operation unit 320 includes an interest detection unit 321 and an interest notification unit 322. The interest detection unit 321 detects the interest of the participant P based on the input operation of the participant P, etc. The interest notification unit 322 notifies the guidance terminal 200 of the interest information indicating the detected interest via the communication unit 340.

[0086] Figure 7 Shows a structural example of the participant screen 310 displayed on the display panel 301 of the participant terminal 300 according to this embodiment. As Figure 7 shown, for example, the participant screen 310 includes a robot image display area 311, a map display area 312, a self-participant image display area 313, an other-participant image display area 314, and a guide image display area 315. In addition, Figure 7 the display of each area in [] is an example and is not limited to these.

[0087] The robot image display area 311 displays the captured image around the robot 100 captured and transmitted by the robot 100. In the map display area 312, the map information and position information around the robot 100 transmitted by the robot 100 are displayed. The map display area 312 displays, for example, information such as tourist attractions around the robot 100, the current location of the robot 100, the moving path, the destination, the orientation, etc.

[0088] The self-participant image display area 313 displays the captured image (self-participant image) mainly including the face of the self-participant P operating the participant terminal 300 captured by the camera 305. The other-participant image display area 314 displays the captured image (other-participant image) mainly including the face of the other participant P operating the other participant terminal 300 captured and transmitted by the other participant terminal 300. The guide image display area 315 displays the captured image (guide image) mainly including the face of the guide G captured and transmitted by the guide terminal 200.

[0089] Figure 8 Shows an operation example of the remote robot system 10 according to this embodiment. As Figure 8 shown, the guide terminal 200 sends an operation signal for remotely operating the robot 100 to the robot 100 according to the operation of the guide G (S101). In the guide terminal 200, when the guide G operates the operation panel displayed in the operation panel display area 215 of the guide screen 210, the arithmetic unit 220 generates an operation signal representing control such as movement and imaging according to the operation of the guide G, and sends the operation signal to the robot 100 via the communication unit 240.

[0090] Next, the robot 100 performs a movement action and a camera action according to the operation signal (S102), and sends the position information and the captured image to the guidance terminal 200 and the plurality of participant terminals 300 (S103). In the robot 100, when the communication unit 170 receives an operation signal from the guidance terminal 200, the control unit 150 controls the driving of the base driving unit 142 and the upper body driving unit 143 according to the received operation signal, and controls the camera action of the camera 131, etc. The control unit 150 sends the captured image captured by the camera 131 and the position information (latitude, longitude, etc.) detected by the GPS unit 135 to the guidance terminal 200 and the plurality of participant terminals 300 via the communication unit 170. In addition, as needed, the map information, sound, etc. around the robot 100 are sent to the guidance terminal 200 and the plurality of participant terminals 300.

[0091] In addition, the position information and the captured image can be sent from the robot 100 to the guidance terminal 200 and the plurality of participant terminals 300 respectively, or the position information and the captured image can be sent from the robot 100 to the guidance terminal 200, and the position information and the captured image are transmitted from the guidance terminal 200 to the plurality of participant terminals 300.

[0092] Next, the guidance terminal 200 and the plurality of participant terminals 300 output the information received from the robot 100 (S104). In the guidance terminal 200, when the communication unit 240 receives the position information and the captured image from the robot 100, the arithmetic unit 220 (display control unit 222) displays the captured image of the robot 100 received in the robot image display area 211 of the guidance screen 210, and displays the position information of the robot 100 received in the map display area 212. In addition, when the guidance terminal 200 receives map information from the robot 100, it updates the map information in the map display area 212, and outputs sound from the speaker 204 when it receives a sound signal.

[0093] Similarly to the guidance terminal 200, in the participant terminal 300, when the communication unit 340 receives the position information and the captured image from the robot 100, the arithmetic unit 320 displays the captured image of the robot 100 received in the robot image display area 311 of the participant screen 310, and displays the position information of the robot 100 received in the map display area 312. In addition, when the participant terminal 300 receives map information from the robot 100, it updates the map information in the map display area 312, and outputs sound from the speaker 304 when it receives a sound signal.

[0094] Next, multiple participant terminals 300 detect the interests and concerns of each participant P based on the operations of each participant P, etc. (S105), and send interest and concern information indicating the detected interests and concerns to the guidance terminal 200 (S106). The interest and concern detection unit 321 of the participant terminal 300 detects the interests and concerns of the participant P based on, for example, the operation state, vocalization state, and gaze state of the participant P. It can be any one of the operation state, vocalization state, gaze state, etc., or any multiple states can be used. When the interest and concern detection unit 321 detects an interest and concern, the interest and concern notification unit 322 sends the information of the detected interest and concern (each state) to the guidance terminal 200 via the communication unit 340.

[0095] For example, the interest and concern detection unit 321 detects the operation state of the participant P with respect to the display (output) of the participant screen 310, and detects the interests and concerns of the participant P based on the detected operation state. For example, the interest and concern notification unit 322 may also send interest and concern information including the detected operation state. The detected operation state includes operation elements such as the operation position, operation times, and operation time in the operation of the participant P on the participant screen 310 where the information of the robot 100 is displayed. Any one of the operation elements can be used, or any multiple operation elements can be used. The detected operation can be a touch operation on the touch panel of the display panel 301 or the movement and click operation of the mouse pointer implemented by using a mouse operation.

[0096] When detecting the operation position, either a specific position operated on the participant screen 310 can be detected, or a predetermined area including the operated position can be detected. For example, the participant screen 310 can also be divided into several areas, and the divided area including the operated position can be detected. The interest and concern detection unit 321 detects the position and area operated in the robot image display area 311 and the map display area 312.

[0097] In addition, when detecting the operation times, the number of touches during a predetermined period, the number of clicks of the mouse, etc. can also be detected. When detecting the operation time, the touch duration (long press time), click duration, etc. can also be detected. For example, the interest and concern detection unit 321 can also detect that the interest and concern in the corresponding area is high (excited) and send it when the number of touches is larger or the touch duration is longer than a predetermined threshold in a predetermined area.

[0098] In addition, the interest attention detection unit 321 detects the vocalization state of the participant P with respect to the display of the participant screen 310, and detects the interest attention of the participant P based on the detected vocalization state. The interest attention notification unit 322 may also send interest attention information including the detected vocalization state. The detected vocalization state includes vocalization elements such as the sound pressure of the sound emitted by the participant P, the vocalization time, and the identification result (keyword) of the sound. Any one of the vocalization elements may be used, or any plurality of vocalization elements may be used.

[0099] The interest attention detection unit 321 detects, for example, the sound pressure of the sound of the participant P through the microphone 303. It is also possible to detect the sound pressure of the sound emitted by the participant P through the microphone 303 and detect the excitement level such as "Wow!" or "Amazing". In the case of detecting the vocalization time, it is also possible to detect the duration for which a predetermined sound pressure persists. For example, the interest attention detection unit 321 may also detect that the interest attention of the participant P has increased when the sound pressure of the participant P is greater than a predetermined threshold or when a predetermined sound pressure persists for a predetermined time.

[0100] In addition, the interest attention detection unit 321 may also detect a predetermined sound through voice recognition. For example, it is also possible to perform voice recognition on the sound of the participant P detected by the microphone 303, and detect that the interest attention of the participant P has increased when a predetermined sound indicating excitement level such as "Wow!" or "Amazing" is recognized. In this case, it is preferable to recognize a predetermined keyword associated with the interest attention. For example, it may also be a keyword associated with the name of a building or a tourist attraction around the robot 100. In addition, for the interest attention (excitement level), not limited to positive keywords, it is also possible to detect negative keywords. Furthermore, in the case of remote communication on the participant terminal 300, it is also possible to detect the vocalization state for interest attention in a state where the sound of the remote communication is muted.

[0101] In addition, the interest attention detection unit 321 may also detect the gaze state of the participant with respect to the display of the participant screen 310, and detect the interest attention of the participant P based on the detected gaze state. The interest attention notification unit 322 may also send interest attention information including the detected gaze state. The detected gaze state includes gaze elements such as the orientation of the face of the participant P, the orientation of the line of sight of the participant P, the gaze position, and the gaze time. Any one of the gaze elements may be used, or any plurality of gaze elements may be used.

[0102] The interest and attention detection unit 321 detects the orientation of the face of the participant P through the image of the camera 305, for example, and sets up a line-of-sight sensor to detect the orientation of the line of sight of the participant P. The interest and attention detection unit 321 infers the fixation position (which can also be a fixation area) on the participant screen 310 based on the orientation of the face and the line of sight of the participant P. When detecting the fixation time, it is also possible to detect the duration for which a predetermined face, line-of-sight orientation, and predetermined fixation position persist. For example, the interest and attention detection unit 321 can also detect that the interest and attention of the participant P at that position (orientation) increases when the same face, line-of-sight orientation, and fixation position persist for a predetermined time. Additionally, the interest and attention detection unit 321 can detect the facial expression of the participant P through image recognition, not limited to the orientation of the face of the participant P. For example, it is also possible to detect a positive expression or a negative expression.

[0103] Next, the guidance terminal 200 displays the interest and attention information received from the multiple participant terminals 300 (S107). In the guidance terminal 200, when the communication unit 240 receives the interest and attention information from the multiple participant terminals 300, the interest and attention analysis unit 221 analyzes the interest and attention of the multiple participants P based on the received interest and attention information, and the display control unit 222 displays the analysis result on the guidance screen 210. The interest and attention analysis unit 221 performs summation processing and statistical processing on the operation state, vocalization state, fixation state, etc. of the participant P, detects the part on the screen that the participant P is fixating on, or detects the excited state of the participant P, and displays the detection result in a manner that enables the guidance G to recognize it. The statistical processing includes processing such as summation values, average values, median values, maximum values, and minimum values, including statistics for each position (area) of the screen and hourly statistics. For example, on the guidance screen 210 for displaying information about the robot 100, it is possible to either emphasize and display the position corresponding to the interest and attention information or display the temporal change of the interest and attention information. As an example of emphasizing and displaying the position, it is also possible to overlay and display a heat map in the robot image display area 211 of the guidance screen 210, and as an example of displaying the temporal change, it is also possible to display a rank table or a graph in the interest and attention display area 216 of the guidance screen 210. It is preferably displayed in a style corresponding to the degree (level) of interest and attention indicated by the interest and attention information. For example, the degree of interest and attention is a value based on the number of operations or operation time in the operation state of multiple participants, the sound pressure or vocalization time in the vocalization state, the fixation time in the fixation state, etc., and it is also possible to weight each element. In addition, it is possible to centrally display the statistical results of all participants, and it is also possible to visibly display the statistical results of each participant. It is also possible to visibly display the number of participants indicating interest and attention.

[0104] Figure 9This is an example of using a heat map to display the analysis results of interest focus information. As Figure 9 shown, the display control unit 222 displays a heat map based on the interest focus information, for example, in the robot image display area 211 of the wizard screen 210. The heat map can be displayed either on the captured image in the robot image display area 211 or on the map information in the map display area 212. The distribution of interest focus is visually displayed in such a way that it overlaps with the part (exciting part) where the gaze of the participant P is concentrated in the captured image or map information. For example, the area with high interest focus (where the interest focus is concentrated) is displayed in red, and the shade and color are changed according to the level of gaze (interest focus) for display.

[0105] Figure 10 This is an example of using a graphic such as a circle to display the analysis results of interest focus information. As Figure 10 shown, the display control unit 222 displays a circular frame based on the interest focus information, for example, in the robot image display area 211 of the wizard screen 210. The circular frame can be displayed either on the captured image in the robot image display area 211 or on the map information in the map display area 212. The circular frame is displayed in such a way that it surrounds the part (exciting part) where the gaze of the participant P is concentrated in the captured image or map information. For example, the area with high interest focus (where the interest focus is concentrated) is displayed with a red frame, and the color, size, thickness of the frame, etc. are changed according to the level of gaze (interest focus) for display. In addition, it is not limited to a circle and can be any other arbitrary shape.

[0106] Figure 11A And Figure 11B This is an example of using a rating scale to display the analysis results of interest focus information. As Figure 11A And Figure 11B shown, the display control unit 222 displays a rating scale corresponding to the interest focus information in the interest focus display area 216 of the wizard screen 210, for example. In the rating scale, the level (high or low) of the current interest focus (excitement level) is displayed. As Figure 11A shown, when the current interest focus is low, a low level is shown in the rating scale, and as Figure 11B shown, when the current interest focus is high, a high level is shown in the rating scale. In the rating scale, the color can also be changed according to the level for display.

[0107] Figure 12 This is an example of using a graphic to display the analysis results of the analyzed interest focus information. As Figure 12 shown, the display control unit 222 displays a graphic corresponding to the interest focus information in the interest focus display area 216 of the wizard screen 210, for example. In the graphic, the temporal change of the excitement level is displayed. AsFigure 12 As shown, the level of interest is displayed as a line graph according to the time series to show the relative change in excitement. It is not limited to a line graph, and it can also be a bar graph or other graphs. In the graph, it can also be displayed by changing the color according to the change in level and time.

[0108] As Figure 8 shown, next, the guide terminal 200, in the same manner as S101, according to the operation of the guide G, sends an operation signal for remotely operating the robot 100 to the robot 100 (S108). The guide G grasps the interest of the participant P according to the interest information displayed on the guide screen 210. For example, the guide G operates the control panel display area 215 of the guide screen 210 in such a way as to move in the direction of higher interest. The arithmetic unit 220 generates an operation signal according to the operation of the guide G and sends the operation signal to the robot 100 via the communication unit 240. After that, the robot 100, in the same manner as S102, performs a movement action, a shooting action, etc. according to the operation signal.

[0109] As described above, in the present embodiment, in a remote robot system capable of implementing remote sightseeing tours, etc., it has a function of detecting the interest of the participant, and also has a function of prompting the detected result to the connection terminal of the guide. Thus, even if the guide and the participant are far apart, it is possible to feedback to the guide where the participant is looking and what the participant is interested in. Therefore, the guide can explain the parts of interest or move the robot to obtain detailed information. Therefore, it is possible to implement a tour according to the expectations of the participant while grasping the reactions of the participant (user), etc., so that it is possible to appropriately respond to the participant.

[0110] (Embodiment 2)

[0111] Next, Embodiment 2 of the present invention will be described. In the present embodiment, an example of automatically controlling the action of the robot according to the interest of the participant in the remote robot system of Embodiment 1 will be described.

[0112] Figure 13 is a block diagram showing a functional structure example of the guide terminal 200 according to the present embodiment. As Figure 13 shown, the guide terminal 200 has a control method determination unit 223 in the arithmetic unit 220. Other structures are the same as those in Embodiment 1. The control method determination unit (remote control determination unit) 223 determines the control method (remote operation method) of the robot 100 according to the interest information notified from the participant terminal 300, that is, according to the interest of the participant P analyzed by the interest analysis unit 221.

[0113] Figure 14Shows an operation example of the remote robot system 10 according to this embodiment. In Figure 14 S101 to S107 are the same as those in Embodiment 1. Next, in S107, the guidance terminal 200 determines a control method for the robot 100 (S110) based on the interest and attention information of the participant P, and sends an operation signal of the determined control method to the robot 100 (S108). When the interest and attention analysis unit 221 analyzes the interest and attention information of the participant P and the display control unit 222 displays the analysis result, the control method determination unit 223 determines a control method for the robot 100 based on the analyzed interest and attention information of the participant P. In addition, the display of the interest and attention information may be omitted, and the control method may be determined in S110.

[0114] For example, the control method determination unit 223 determines a movement method for the robot 100 including the movement direction, movement path, movement speed, etc. of the robot 100. The control method determination unit 223 may also automatically select a path with high interest from multiple tour paths based on the analyzed interest and attention information. For example, when there are many participants P observing a specific building, a path passing near the building is selected. In addition, the control method determination unit 223 may control the movement speed of the robot 100 based on the analyzed interest and attention information. For example, the speed of the robot 100 may be adjusted according to the degree of interest, the appreciation time of the venue with interest may be extended, and the places with low interest may be controlled to be shorter. In a place with high interest, the speed may be reduced or the movement may be stopped.

[0115] In addition, the control method determination unit 223 may also determine an information acquisition method for the robot 100 including the imaging direction of the image of the robot 100, the sound collection direction, etc. The control method determination unit 223 may either control the imaging direction of the camera 131 of the robot 100 to automatically face the direction of interest based on the analyzed interest and attention information, or magnify the imaging range of the camera 131. Similarly, the sound collection direction of the microphone 132 may be controlled to face the direction of interest.

[0116] In addition, the robot 100 may be automatically controlled by sending an operation signal in accordance with the determined control method, or the determined control method may be displayed on the guidance screen 210 to recommend the movement path, etc. of the robot 100 to the guide G. In addition, the determination of the control method and the execution of the control are not limited to the guidance terminal 200, but may also be performed by other devices.

[0117] As described above, in this embodiment, in the remote robot system, the control method of the robot is determined based on the interest and attention of the participants. Thereby, the robot can be automatically controlled according to the interest and attention of the participants, and the participants can be appropriately responded to.

[0118] In addition, the present invention is not limited to the above-described embodiments and can be appropriately modified without departing from the gist. For example, in the above-described embodiments, an example of using a remote robot system for remote sightseeing tours has been described, but it is not limited to remote sightseeing tours and can also be used for other purposes. For example, it can also be used in cases where missing persons such as missing people are remotely searched or in cases where the status of family members, etc. is remotely confirmed. In the case where a robot is remotely operated by multiple remote users, when it is not known where to focus for remote operation, the above-described embodiments can be effectively utilized.

[0119] Each structure in the above-described embodiments is composed of hardware or software, or both. It can be composed of one piece of hardware or software, or multiple pieces of hardware or software. The functions (processing) of each device can also be implemented by a computer having a CPU, a memory, etc. For example, each function can also be implemented by storing a program for performing the method (control method) in the above-described embodiments in a storage device and having the CPU execute the program stored in the storage device.

[0120] These programs include a group of commands (or software codes) that, when read by a computer, cause the computer to perform one or more functions described in the embodiments.

[0121] The program can also be stored in a non-temporary computer-readable medium or a certain storage medium of an entity. Without being limited thereto, as an example, a computer-readable medium or a certain storage medium of an entity includes a random access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive (SSD), or other memory technologies, a CD-ROM, a digital versatile disc (DVD), a Blu-ray (registered trademark in Japan) disc, or other optical disc storage devices, a magnetic cassette, a magnetic tape, a magnetic disk storage device, or other magnetic storage device apparatuses. The program can also be sent to a temporary computer-readable medium or a communication medium. Without being limited thereto, as an example, a temporary computer-readable medium or a communication medium includes a transmission signal in an electrical form, an optical form, a sound form, or other forms.

[0122] From the disclosure thus described, it is obvious that the embodiments of the present disclosure can vary in many ways. Such variations should not be regarded as departing from the spirit and scope of the present disclosure, and all such modifications that are obvious to those skilled in the art are intended to be included within the scope of the appended claims.

Claims

1. A remote robot system, comprising: A robot that performs predetermined actions including collecting local information around it; A first remote terminal that generates an operation signal for moving and photographing the robot based on an accepted operation and can remotely operate the actions of the robot; and A second remote terminal that can communicate remotely with the first remote terminal, The second remote terminal includes: An information output unit that outputs the local information obtained from the robot to the user; and An interest detection unit that detects the user's interest in the output local information, The user associated with the second remote terminal conducts a remote sightseeing tour via the robot remotely operated by the first remote terminal, The second remote terminal includes a notification unit that notifies the interest information indicating the detected interest to the first remote terminal, The first remote terminal includes an interest output unit that overlays and displays a heat map indicating the position corresponding to the interest information on the display screen where the local information is displayed, and displays by changing the shade and color of the heat map according to the level of the interest.

2. The remote robot system according to claim 1, wherein The interest detection unit detects the operation state of the user with respect to the output local information, and detects the user's interest based on the detected operation state.

3. The remote robot system according to claim 2, wherein The operation state includes the operation position, the number of operations, or the operation time in the operation of the user on the display screen where the local information is displayed.

4. The remote robot system according to any one of claims 1 to 3, wherein The interest detection unit detects the vocalization state of the user with respect to the output local information, and detects the user's interest based on the detected vocalization state.

5. The remote robot system according to claim 4, wherein The vocalization state includes the sound pressure, the vocalization time, or the recognition result of the sound emitted by the user.

6. The remote robot system according to any one of claims 1 to 3, wherein The interest detection unit detects the gazing state of the user with respect to the output local information, and detects the user's interest based on the detected gazing state.

7. The remote robot system according to claim 6, wherein The gazing state includes the orientation of the user's face, the orientation of the user's line of sight, the gazing position, or the gazing time.

8. The remote robot system according to claim 1, wherein The interest output unit displays the temporal change of the interest information on the display screen where the local information is displayed.

9. The remote robot system according to claim 8, wherein The first remote terminal includes a remote control determination unit that determines a remote operation method of the robot based on the notified interest information.

10. The remote robot system according to claim 9, wherein the remote control determination unit determines a movement method of the robot including a movement direction, a movement path, or a movement speed of the robot based on the interest information.

11. The remote robot system according to claim 9 or 10, wherein the remote control determination unit determines an information acquisition method of the robot including a shooting direction of an image of the robot or a sound collection direction of sound based on the interest information.

12. A control method of a remote robot system, wherein the remote robot system includes: a robot that performs a predetermined action including collection of local information around; a first remote terminal that generates an operation signal for moving and shooting the robot according to an accepted operation and can remotely operate the action of the robot; and a second remote terminal that can remotely communicate with the first remote terminal, in the control method of the remote robot system, the second remote terminal outputs the local information obtained from the robot to a user, the second remote terminal detects an interest of the user in the output local information, the user associated with the second remote terminal performs a remote sightseeing tour via the robot remotely operated by the first remote terminal, the second remote terminal notifies the first remote terminal of interest information indicating the detected interest, the first remote terminal overlays and displays a heat map indicating a position corresponding to the interest information on a display screen on which the local information is displayed, and displays by changing the shade and color of the heat map according to the level of the interest.

Images