Communication terminal, remote control method, and program

Abstract

To suppress operations performed by a plurality of users in a mixed manner even when the plurality of users remotely control a remote-controlled moving device.SOLUTION: The present disclosure relates to a communication terminal capable of remotely controlling a remote-controlled moving device. The communication terminal includes: a reception unit that accepts remote control of an operation target of the remote-controlled moving device; and a transmitting unit capable of remotely controlling the operation target by transmitting operation information indicating the remote control of the operation target, which is an application target of an exclusive operation right.SELECTED DRAWING: Figure 20

Description

Technical Field

[0001] The present disclosure relates to a communication terminal, a communication management system, a remotely operated mobile device, a communication system, a remote operation method, a communication management method, a control method, a communication method, and a program.

Background Art

[0002] Communication systems in which communication terminals communicate between multiple bases via a communication network such as the Internet or a LAN (Local Area Network) to transmit and receive video and / or audio to and from each other are widespread. By using this communication system, users at bases in remote locations can communicate with each other, for example, hold a remote meeting.

[0003] Also, a telepresence robot (hereinafter referred to as "robot") that combines technologies of remote conferencing, robots, and remote operation is known (see Patent Document 1). This is a technology that combines remote control from a base in a remote location and robot technology, allowing a user to operate a robot from a base in a remote location and behave as if the user were present (present) at a certain location.

[0004] The robot usually has a display and has the functions of a remote conferencing device using it and the ability to move by remote operation. While a user conducts a remote meeting using a PC (Personal Computer), smartphone, or tablet terminal, the user can control the robot on the other side from a base in a remote location, thereby controlling who to communicate with and where to go as the main body. Different from conventional remote conferencing technologies, the robot can provide an unprecedented sense of presence to the counterpart in a remote location by moving to another location or interacting as a substitute of the user himself / herself.

[0005] Furthermore, there are facilities such as museums and art galleries that display valuable objects, academic materials, and works of art to visitors in the form of exhibits. In response to the decline in visitors due to the recent restrictions on going out caused by COVID-19, services such as online experiential tours have begun to be offered. There is also a movement to introduce such services in corporate showrooms.

[0006] In these types of services, pilot projects are underway to introduce remote tours using robots, allowing users in remote locations to freely tour facilities online. These remote tours provide multiple users in remote locations with a simulated experience of visiting the facility by allowing them to view video footage captured by robots. Furthermore, users can also remotely control the robots to freely explore the facility. [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, while this assumes that one user has exclusive control of a remotely operated mobile device such as a robot, the problem arises as to how to manage remote control of the device when multiple users are viewing footage captured by the device, such as in a remote tour. Currently, when one user tries to operate a remotely operated mobile device, another user can access it even if the device is already being operated, resulting in a situation where various users are operating the device simultaneously. This can lead to situations such as, for example, one user remotely controlling a remotely operated mobile device to move it to the right towards its target location, while another user remotely controls it to move it to the left, causing it to move to a different target location.

[0008] This invention has been made in view of the above-mentioned problems, and aims to suppress operations performed by multiple users in a chaotic manner, even when multiple users remotely operate a remotely controlled mobile device. [Means for solving the problem]

[0009] In view of the above problems, the invention according to claim 1 is a communication terminal that can be remotely controlled with respect to a remotely controlled mobile device, wherein the remotely controlled mobile device multiple A reception unit that accepts remote control requests for the target object, different The above is subject to the right of operation. multiple By transmitting operation information indicating the remote operation on the target of the operation, multiple The target can be controlled remotely. Let's assume The transmitting unit, A display control unit that displays a screen associated with the communication terminal or the user of the communication terminal to which the current operating rights are set, for each of the multiple operating targets of the remotely operated mobile device, immediately after the start of video distribution or in response to a predetermined operation during video distribution, This is a communication terminal characterized by having [a certain feature]. [Effects of the Invention]

[0010] According to the present invention, even when multiple users remotely operate a remotely controlled mobile device, it is possible to suppress operations performed by multiple users in a chaotic manner. [Brief explanation of the drawing]

[0011] [Figure 1] This is an example of a diagram illustrating the general outline of a communication system. [Figure 2] This diagram illustrates an example of displaying a 360-degree image captured by a wide-angle camera. [Figure 3] This figure shows an example configuration of a communication system according to one embodiment. [Figure 4] This figure shows an example of the hardware configuration of a communication terminal according to one embodiment. [Figure 5] This figure shows an example of the hardware configuration of a communication management system. [Figure 6] This figure shows an example of the hardware configuration of a mobile device. [Figure 7] This is a hardware configuration diagram of an example of a wide-angle imaging device. [Figure 8] This is an example of a diagram illustrating the use of a wide-angle photographic device. [Figure 9] This diagram illustrates the general process for creating equirectangular projection images and 360-degree images from images captured by a wide-angle imaging device. [Figure 10] It is a diagram for explaining an outline of processing until an orthographic cylindrical projection image and an omnidirectional image are created from an image captured by a wide-angle imaging device. [Figure 11] It is a diagram showing the positions of a virtual camera and a predetermined area when the omnidirectional image is a three-dimensional spherical object. [Figure 12] It is a diagram showing an example such as a three-dimensional perspective view of FIG. 11. [Figure 13] It is an example of a diagram showing the relationship between predetermined area information and an image of a predetermined area. [Figure 14] It is a functional block diagram of each device or system for constructing a communication system. [Figure 15] It is a functional block diagram of each device or system for constructing a communication system. [Figure 16] It is a sequence diagram showing an example of processing in a communication preparation stage of a communication system. [Figure 17] It is a diagram showing an example of a destination selection screen displayed on a communication terminal. [Figure 18] It is a sequence diagram showing an example of communication processing of a communication system. [Figure 19] It is a diagram showing a display screen of operation right information. [Figure 20] It is a sequence diagram showing processing (part one) for remotely operating a telepresence robot. [Figure 21] It is a sequence diagram showing processing (part two) for remotely operating a telepresence robot. [Figure 22] It is an image diagram applying this embodiment to an online tour of an art museum. [Figure 23] It is an example of a screen displayed on communication terminal 10A during an online tour of an art museum. [Figure 24] It is an image diagram applying this embodiment to remote shopping. [Figure 25] It is an example of a screen displayed on communication terminal 10A during remote shopping.

Embodiments of the Invention

[0012] [Outline of the communication system] The general outline of communication system 1 will be explained using Figure 1. Figure 1(a) shows an example of a schematic configuration diagram of a communication system using the telepresence robot (hereinafter referred to as "robot") R.

[0013] Robot R is mainly composed of a wide-angle camera 9, a communication terminal 10X, and a mobile device 20 for moving Robot R. Robot R is an example of a remotely operated mobile device. A remotely operated mobile device is a device that moves on land, in the air, in space, on water, underwater, or underground by remote control, and also includes autonomous mobile devices. Remotely operated mobile devices include remotely operated mobile devices equipped with driving capabilities. Autonomous mobile devices include autonomous mobile devices equipped with driving capabilities. Robot R is a type of remotely operated mobile device or autonomous mobile device. Robot R is also equipped with environmental sensors such as a temperature sensor, humidity sensor, oxygen sensor, or carbon dioxide sensor, and is also equipped with lighting devices to illuminate the area around Robot R.

[0014] For example, a robot R is located in the company building 801, and a user, operator 810, is located at home 802. Robot R has a communication terminal 10X and a wide-angle camera 9 mounted on a mobile device 20. User a, who could be a remote operator, has a communication terminal 10A. User b, who could be a remote operator, has a communication terminal 10B. User c, who could be a remote operator, has a communication terminal 10C.

[0015] Furthermore, while Figure 1(b) shows three users and three communication terminals, there may be four or more users and four or more terminals. Alternatively, there may be one user and one communication terminal, or two users and two communication terminals.

[0016] The wide-angle shooting device 9 transmits a 360-degree image (an example of a second image), described later, to the communication terminal 10X. Communication terminals 10A, 10B, and 10C each have a narrow-angle shooting unit 14 and a microphone, described later, and transmit and receive video and audio from each other. Communication terminal 10X transmits the 360-degree image and at least one of the front image 831 and rear image 832 to communication terminals 10A, 10B, and 10C, and communication terminals 10A, 10B, and 10C transmit the front image 831 of each user a, b, and c to communication terminal 10X.

[0017] Furthermore, application software (hereinafter referred to as "app") that accepts operations related to the movement of the mobile device 20 is running on communication terminals 10A, 10B, and 10C. Each user a, b, and c inputs operation instructions while viewing the video transmitted by communication terminal 10X. Operation information indicating the remote operation (operation instructions) entered by operator 810 is transmitted from communication terminal 10A to communication terminal 10X via the communication network 2.

[0018] The communication terminal 10X and the mobile device 20 can communicate using short-range wireless technology such as Bluetooth®. The communication terminal 10X controls the mobile device 20 based on operational instructions from a specific user who has control rights, allowing the specific user to move the mobile device 20 from a remote location (home 802).

[0019] The right of operation is the exclusive right (exclusive right of operation) that a specific user (including a specific communication terminal) among multiple users (including communication terminals) has to remotely control each control target (narrow-angle imaging unit 14, mobile device 20, etc.) of robot R. Different rights of operation are set for each control target. The right of operation can also be described as a special right that allows a specific user to perform remote control. In other words, users who do not have the right of operation cannot remotely control any control target of robot R. Note that users include not only people but also organizations such as departments and companies.

[0020] Furthermore, if the narrow-angle imaging device built into the communication terminal 10X can only capture images in front or behind, it would be difficult for the user to check the surroundings of the mobile device 20 (for example, the area around their feet or to the left and right). Therefore, the communication terminal 10X is equipped with a wide-angle imaging device 9 that can capture images of 360 degrees around it. The wide-angle imaging device 9 may be mounted on the communication terminal 10X or on the mobile device 20. Because the wide-angle imaging device 9 can capture images of a wide angle range, it is positioned, for example, above the robot R where the surroundings are less likely to be obstructed. In Figure 1, it is positioned above the communication terminal 10X, but it may be positioned on the left or right side, or on the pole portion of the mobile device 20. Also, the wide-angle imaging device 9 may be built into the communication terminal 10X or attached externally.

[0021] [Display of 360-degree image] Using Figure 2, we will explain an example of a display that allows the user to understand the direction the mobile device 20 is facing when they are looking in a direction on a 360-degree spherical image that captures subjects in all directions. Figure 2 is a diagram illustrating an example of a display of a predetermined area image, which is a part of the 360-degree spherical image captured by the wide-angle shooting device 9. In this embodiment, we will explain three patterns of display examples that allow the user (for example, user a) to understand the direction the mobile device 20 is facing.

[0022] Since the orientation of the mobile device 20 is the same as the orientation of the robot R, the mobile device 20 and the robot R can be considered identical in terms of orientation. The orientation of the mobile device 20 refers to the direction in which the mobile device 20 is facing, for example, the front direction or the forward direction. If the user can understand the direction in which the mobile device 20 is facing, the orientation of the mobile device 20 may also be sideways, upward, downward, etc.

[0023] Figure 2(a) shows an example of a 360-degree video display screen 1010 that displays a 360-degree video using Pattern 1. Pattern 1 allows the user to understand the orientation of the mobile device 20 by overlaying an image 840 of robot R onto the 360-degree video currently being viewed by the user. The user can then see which direction they are looking in relation to the orientation of the mobile device 20. Note that the image 840 of robot R is displayed in a manner that allows the user to see the orientation of robot R.

[0024] The user displays a predetermined area of ​​video on the communication terminal 10A. The video displayed on the 360-degree video display screen 1010 is the predetermined area video. In Figure 2(a), a portion of an open living room is displayed. The predetermined area video is a portion of the 360-degree video displayed on the display device of the communication terminal 10A (display unit 109, described later). The display unit 109 provides an area, and the user displays the predetermined area video within this area. Because the subject of the 360-degree video is displayed with a curved appearance (see Figure 9(c)), when a predetermined area video (see Figures 12(b) and (d)), which is a portion of the 360-degree video, is displayed, it becomes easier for the user to view.

[0025] Furthermore, as described above, the direction does not necessarily coincide with the front direction of the mobile device 20. The communication terminal 10A rotates the robot R based on the difference between the front direction of the mobile device 20 and the imaging direction that defines the predetermined area image (the direction indicating the center of the predetermined area image), and displays the rotated robot R in the 360-degree image. The displayed image 840 of the robot R is not the robot R in real space but is created using CG (computer graphics). CG is an image drawn using a computer. CG mainly comes in 2D and 3D, but in this embodiment, 3D CG is used.

[0026] The direction in which robot R (mobilization device 20) is facing can be seen within the 360-degree image. In the example in Figure 2(a), robot R is facing slightly to the right of user a. By looking at image 840 of robot R, the operator can easily determine which direction it will move if it is moved forward in this position.

[0027] Note that in Figure 2(a), for the sake of explanation, the entire image 840 of robot R is shown. However, since the 360-degree video is positioned above robot R, it is preferable to display image 840 of robot R closer to the viewpoint (in front). For this reason, the entire image 840 of robot R does not need to be shown.

[0028] Next, we will explain Pattern 2. Figure 2(b) shows an example of a 360-degree image display screen 1010 that displays a 360-degree image using Pattern 2. In Pattern 2, in addition to a predetermined area image 853 of the 360-degree image that the operator 810 is viewing, the direction that the operator 810 is looking at is displayed with respect to the robot R. Specifically, it displays an image 854 of the robot R and an arrow 8 indicating the imaging direction.

[0029] As shown in Figure 2(b), the communication terminal 10A displays an arbitrary predetermined area image 853 in the 360-degree image display 1001 in response to user a's operation. In Figure 2(b), displayed goods are shown. In addition to the 360-degree image display 1001, the communication terminal 10A also displays a display 1002 for robot R. The display 1002 for robot R displays an imaging direction arrow 8 indicating the direction of the predetermined area image 853 from the robot R's perspective. In other words, the imaging direction arrow 8 represents the imaging direction as seen by user a in three dimensions. Three dimensions means that it contains information on azimuth and elevation angles. The displayed image 854 of robot R is not a real robot R but is created using computer graphics. The same applies to the imaging direction arrow 8. In Figure 2(b), the direction in front of robot R is the imaging direction.

[0030] In this way, the user can understand which direction the 360-degree image is from the perspective of robot R. The user can also understand which direction robot R is facing. Furthermore, Figure 2(b) shows that robot R approaches the product by moving forward. Compared to pattern 1, there is the advantage that robot R does not overlap with the 360-degree image.

[0031] Figure 2(c) shows an example of a 360-degree video display screen 1010 that displays a 360-degree video according to pattern 3. The video displayed on the 360-degree video display screen 1010 is a predetermined region video. In pattern 3, the communication terminal 10A displays the operation button 860 only when the center direction of the predetermined region video (the imaging direction mentioned above) and the front direction of the robot R approximately coincide. In Figure 2(c), left figure c-1, the operation button 860 is not displayed because the imaging direction and the front direction of the robot R do not approximately coincide. In Figure 2(c), right figure c-2, the operation button 860 is displayed because the imaging direction and the front direction of the robot R approximately coincide. Approximately coincidence means that the coincidence should not hinder the movement of the mobile device 20. If perfect coincidence is required, the operation button 860 for remotely operating the robot R would rarely be displayed and therefore impossible to operate. Therefore, the judgment is made with a margin that does not reduce operability. For example, a deviation of about ±10 to 20 degrees in the horizontal and elevation directions may be allowed. Furthermore, it is beneficial that the operator 810 can set the magnitude of the misalignment.

[0032] In Pattern 3, the operation button 860 is displayed when the imaging direction and the front direction of robot R are roughly aligned. Therefore, the user can change the image in a predetermined area and determine that the imaging direction and the front direction of robot R are roughly aligned when the operation button 860 is displayed. Thus, the imaging direction can be aligned with the direction of the 360-degree image viewed by the user. Furthermore, since the operation button 860 is displayed only when the front direction of robot R and the direction of the 360-degree image viewed by the user are aligned, the user can understand where the robot will move when it moves forward.

[0033] [System Configuration] Figure 3 shows an example configuration of a communication system according to one embodiment. The communication system 1 includes a plurality of communication terminals (10A, 10B, 10C, 10X), a mobile device 20, a relay device 30, and a communication management system 50, etc. In the following description, "communication terminal 10" will be used to refer to any communication terminal among the plurality of communication terminals (10A, 10B, 10C, 10X). Also, "communication terminal 10α" will be used to refer to any communication terminal among the user-side communication terminals (10A, 10B, 10C). Furthermore, the number of communication terminals 10 in Figure 3 is just an example.

[0034] The communication terminal 10, relay device 30, and communication management system 50 are each connected to other communication terminals 10, devices, and systems via the communication network 2, enabling communication between them. The communication network 2 may include, for example, a LAN (Local Area Network), the Internet, a mobile phone network, or a dedicated line.

[0035] The communication terminal 10 is, for example, a general-purpose information processing device such as a tablet, smartphone, or PC, or a communication terminal such as a dedicated remote conferencing device. The communication terminal 10 can conduct remote conferences, etc., with one or more other communication terminals 10 by, for example, sending and receiving video and / or audio. Remote conferencing is an example of video calls. Video calls are not limited to conducting conferences. Video calls are also an example of video streaming. Video streaming is not limited to conducting calls.

[0036] The communication terminal 10α can conduct remote conferences with the communication terminal 10X by running an application compatible with the communication system 1, and can also remotely control the mobile device 20 via the communication terminal 10X. For example, by operating the operation buttons 860 displayed on the remote conference display screen, the communication terminal 10α can move the mobile device 20, which is equipped with the communication terminal 10X, for example, forward, backward, left, or right. The communication terminal 10α can also run browser software (hereinafter referred to as "browser software"), which can display video and accept commands from the mobile device 20 to the browser software.

[0037] The robot R's mobile device 20 is a device that has a driving function that performs movements such as "forward," "backward," "right turn," and "left turn" by driving multiple wheels in response to control from a communication terminal 10X attached to the mobile device 20. If the user remotely instructs the robot to move left or right, it will move forward after "right turn" or "left turn." Note that the appearance of the mobile device 20 shown in Figure 3 is merely an example. The mobile device 20 only needs to be able to move together with the communication terminal 10X in response to operation instructions from the communication terminal 10X mounted on the mobile device 20.

[0038] The relay device 30 is, for example, an information processing device or a system including one or more information processing devices, and relays content data (including video and sound) transmitted and received between multiple communication terminals 10. This sound includes voice, noise, etc. However, the multiple communication terminals 10 may also transmit and receive content data directly without going through the relay device 30.

[0039] The communication management system 50 is, for example, an information processing device or a system including one or more information processing devices. The communication management system 50 performs tasks such as login authentication from the communication terminal 10, management of the communication status of the communication terminal 10, management of destination lists, and control of sessions that communicate between multiple communication terminals 10 via the relay device 30.

[0040] In one embodiment, a session is realized when the relay device 30 relays content data, including video and / or audio, between multiple communication terminals 10.

[0041] In the above configuration, for example, user a of communication terminal 10A can communicate with communication terminal 10X and remotely move communication terminal 10X and mobile device 20. This allows user a of communication terminal 10A to move communication terminal 10X and mobile device 20 to the vicinity of any other party and conduct remote meetings, etc.

[0042] [Hardware configuration] <Hardware configuration of communication terminals> Figure 4 shows an example of the hardware configuration of a communication terminal according to one embodiment of the present invention. The communication terminal 10 includes the configuration of a general computer, for example, a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, flash memory 104, and SSD (Solid State Drive) 105. The communication terminal 10 also has a media I / F (Interface) 107, an input unit 108, a display unit 109, a network I / F 111, a camera 112, an image sensor I / F 113, a microphone 114, a speaker 115, and an audio input / output I / F 116. Furthermore, the communication terminal 10 has an external device connection I / F 117, a short-range wireless communication unit 118, and a bus 119.

[0043] The CPU 101 is a processing unit that realizes the various functions of the communication terminal 10 by reading programs and data from, for example, the ROM 102 and flash memory 104, and executing processing. The ROM 102 is a non-volatile memory that pre-stores programs used to start the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is a volatile memory used as the work area of ​​the CPU 101.

[0044] The flash memory 104 is a storage device that stores, for example, the OS (Operating System), application software programs, and various types of data. The SSD 105 controls the reading and writing of various types of data to the flash memory 104 according to the control of the CPU 101. The media I / F 107 controls the reading or writing (storage) of data to the recording medium 106, which is a recording medium such as a memory card. The recording medium may store a computer-readable program that controls the communication terminal 10.

[0045] The input unit 108 is an input device for receiving input operations from the operator, such as a touch panel, keyboard, or pointing device. Voice input may also be possible. The display unit 109 is a display device that shows various information to the operator. The input unit 108 and the display unit 109 may also be a display input unit 110, such as a touch panel display in which a touch panel and a display are integrated. Furthermore, the display unit 109 may be not only a display provided on the communication terminal 10, but also an external display of the communication terminal 10, a display of another external communication terminal of the communication terminal 10, or a screen projected by a projector (including the object to be displayed by projection mapping).

[0046] The network interface 111 is a communication interface for the communication terminal 10 to send and receive data using the communication network 2. The camera 112 includes an image sensor for capturing images of a subject according to the control of the CPU 101. The image sensor interface 113 controls the imaging by the camera 112 and converts the captured data into predetermined image data. The camera 112 is a narrow-angle shooting device and captures images with a narrower field of view than the wide-angle shooting device 9, but has a higher resolution than the wide-angle shooting device 9. Resolution is a measure of the fineness of a digital image, and is expressed by quantifying the fineness of the individual unit points (dots, pixels) that make up the digital image. The unit generally used to express resolution is "dots". In the case of displays, it is often expressed as the number of dots arranged horizontally and vertically, such as "1024 x 768 dots". When comparing the same imaging range, the number of dots in the wide-angle shooting device 9 is less than the number of dots in the camera 112.

[0047] Microphone 114 converts recorded audio into audio data. Speaker 115 converts the audio data back into sound and outputs it. Audio input / output interface 116 controls the input and output of audio from microphone 114 and speaker 115.

[0048] The external device connection interface 117 is an interface for connecting external devices, such as USB (Universal Serial Bus). External devices include, for example, the mobile device 20 shown in Figure 3.

[0049] The short-range wireless communication unit 118 is a communication interface for communicating with external devices (e.g., mobile device 20) using short-range wireless communication such as Bluetooth® or Bluetooth® Low Energy. The bus 119 is connected to all of the above configurations in common and transmits address signals, data signals, and various control signals.

[0050] <Hardware configuration of the communication system> Figure 5 shows an example of the hardware configuration of a communication management system according to one embodiment. The communication management system 50 has a general computer configuration, and for example, includes a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, media drive 507, and display 508. The communication management system 50 also includes a network I / F 509, keyboard 511, mouse 512, CD-ROM drive 514, and bus 510.

[0051] The CPU 501 is a computing unit that realizes the various functions of the communication management system 50 by reading programs and data stored in, for example, the ROM 502 and HD 504, and executing processing. The ROM 502 is a non-volatile memory that pre-stores programs used to start the CPU 501, such as the IPL. The RAM 503 is a volatile memory used as the work area of ​​the CPU 501.

[0052] HD504 is a storage device that stores, for example, the OS, application software programs, and various types of data. The HDD controller 505 controls the reading or writing of various types of data to the HD504 according to the control of the CPU 501. The display 508 is a display device that displays various types of information, such as cursors, menus, windows, characters, or images.

[0053] Network I / F 509 is a communication interface for performing data communication using communication network 2. Keyboard 511 is an example of an input device for receiving input operations such as characters, numbers, and various instructions from the system administrator. Mouse 512 is an example of a pointing device for receiving operations such as selecting and executing various instructions, selecting processing targets, and moving the cursor from the system administrator.

[0054] Furthermore, the communication management system 50 or relay device 30 does not necessarily have to be present with respect to the display 508, keyboard 511, and mouse 512. In this case, they may be connected as needed.

[0055] The media drive 507 controls the reading and writing (storage) of data to a recording medium 506, such as a memory card. The CD-ROM drive 514 controls the reading or writing of data to a disk 513, which is an example of a removable recording medium. The bus 510 electrically connects each of the above components and transmits address signals, data signals, and various control signals. Note that the above computer hardware configuration is merely an example.

[0056] <Hardware configuration of the relay device> The relay device 30 shall have the same hardware configuration as the communication management system 50. The programs for the communication terminal 10, the relay device 30, and the communication management system 50 may be distributed as installable or executable files recorded on computer-readable recording media. Examples of such recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray disc, and USB memory. Furthermore, the recording media such as CD-ROMs containing these programs, and the HD504 containing these programs, may be provided domestically or internationally as a program product.

[0057] <Hardware configuration of the mobile device> Figure 6 shows an example of the hardware configuration of a mobile device according to one embodiment. The mobile device 20 includes, for example, a CPU 401, RAM 402, ROM 403, external device I / F 404, short-range wireless communication unit 405, wheel drive unit 406, steering unit 407, etc.

[0058] The CPU 401 is an arithmetic unit that realizes the various functions of the mobile device 20 by executing programs stored in the ROM 403, etc. The RAM 402 is a volatile memory used as a work area for the CPU 401, etc. The ROM 403 is a non-volatile memory that stores programs, etc., of the mobile device 20. The ROM 403 may be a rewritable non-volatile memory such as flash ROM.

[0059] External device I / F404 is a wired communication interface for communicating with the external device connection I / F117, etc., of the communication terminal 10 via a wired connection.

[0060] The short-range wireless communication unit 405 is, for example, a wireless communication interface for performing wireless communication using the same wireless communication method as the short-range wireless communication unit 118 of the communication terminal 10. The mobile device 20 only needs to be able to communicate with the communication terminal 10, for example, via the external device I / F 404 or the short-range wireless communication unit 405.

[0061] The wheel drive unit 406 is an example of a drive device that drives the wheels for moving the mobile device 20. The wheel drive unit 406 includes, for example, a motor.

[0062] The steering unit 407 is an example of a steering device that steers the mobile device 20, which is moved by the wheel drive unit 406. The steering unit 407 may, for example, change the direction or tilt of the wheels, or it may change the direction of the mobile device 20 by controlling the rotation speed and velocity of the left and right wheels.

[0063] <Hardware configuration of wide-angle shooting device> The hardware configuration of the wide-angle shooting device 9 will be explained using Figure 7. Figure 7 is a hardware configuration diagram of an example of the wide-angle shooting device 9. In the following, the wide-angle shooting device 9 is described as a 360-degree (omnidirectional) wide-angle shooting device 9 using two image sensors, but there may be more than two image sensors. Furthermore, it is not necessarily required to be a device dedicated to omnidirectional imaging; an omnidirectional imaging unit can be attached to a regular digital camera or smartphone, etc., to effectively achieve the same functionality as the wide-angle shooting device 9.

[0064] As shown in Figure 7, the wide-angle shooting device 9 is also connected to an imaging unit 301, an image processing unit 304, an imaging control unit 305, a microphone 308, a sound processing unit 309, a CPU 311, a ROM 312, an SRAM 313, a DRAM 314, an operation unit 315, a network interface 316, a communication unit 317, an antenna 317a, an electronic compass 318, a gyro sensor 319, and an acceleration sensor 320, among others.

[0065] Of these, the imaging unit 301 includes wide-angle lenses (so-called fisheye lenses) 302a and 302b, each having a field of view of 180° or more for forming hemispherical images, and two image sensors 303a and 303b, each corresponding to the wide-angle lens. The image sensors 303a and 303b include image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert the optical image from the fisheye lenses 302a and 302b into electrical signal image data and output it, timing generation circuits that generate horizontal or vertical synchronization signals and pixel clocks for these image sensors, and a group of registers in which various commands and parameters necessary for the operation of these image sensors are set.

[0066] The image sensors 303a and 303b of the imaging unit 301 are each connected to the image processing unit 304 via a parallel I / F bus. On the other hand, the image sensors 303a and 303b of the imaging unit 301 are connected separately from the imaging control unit 305 via a serial I / F bus (such as an I2C bus). The image processing unit 304, the imaging control unit 305, and the sound processing unit 309 are connected to the CPU 311 via bus 310. Furthermore, ROM 312, SRAM 313, DRAM 314, operation unit 315, network I / F 316, communication unit 317, and electronic compass 318 are also connected to bus 310.

[0067] The image processing unit 304 receives image data output from image sensors 303a and 303b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create equirectangular projection image data as shown in Figure 9(c).

[0068] The imaging control unit 305 generally uses the I2C bus to set commands and other information in the registers of the image sensors 303a and 303b, with the imaging control unit 305 acting as the master device and the image sensors 303a and 303b as slave devices. It receives necessary commands and other information from the CPU 311. The imaging control unit 305 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 303a and 303b and send it to the CPU 311.

[0069] Furthermore, the imaging control unit 305 instructs the image sensors 303a and 303b to output image data when the shutter button on the operation unit 315 is pressed. Depending on the wide-angle shooting device 9, there may also be functions to support preview display on a display (for example, the display of the communication terminal 10X) and video display. In this case, the image data output from the image sensors 303a and 303b is performed continuously at a predetermined frame rate (frames / minute).

[0070] As will be described later, the imaging control unit 305 also functions as a synchronization control means that works in cooperation with the CPU 311 to synchronize the output timing of image data from the image sensors 303a and 303b. In this embodiment, the wide-angle shooting device 9 is not provided with a display, but a display unit may be provided.

[0071] Microphone 308 converts the recorded audio into audio data. The sound processing unit 309 receives the audio data output from microphone 308 via the I / F bus and performs predetermined processing on the audio data.

[0072] The CPU 311 controls the overall operation of the wide-angle imaging device 9 and performs necessary processing. The ROM 312 stores various programs for the CPU 311. The SRAM 313 and DRAM 314 are work memories that store programs executed by the CPU 311 and data in progress. In particular, the DRAM 314 stores image data in progress and processed equirectangular projection image data from the image processing unit 304.

[0073] The control unit 315 is a general term for the operation buttons, such as the shutter button. The operator uses the control unit 315 to input various imaging modes, imaging conditions, and other settings.

[0074] The network interface 316 is a general term for interface circuits (such as USB interfaces) that connect to external media such as SD cards or personal computers. The network interface 316 can be wireless or wired. The data of the equirectangular projection image stored in the DRAM 314 is recorded to external media via this network interface 316, or transmitted to external terminals (devices) such as the communication terminal 10X via the network interface 316 as needed.

[0075] The communication unit 317 communicates with an external terminal (device) of the communication terminal 10X via an antenna 317a provided on the wide-angle imaging device 9 using short-range wireless communication technologies such as Wi-Fi, NFC, and Bluetooth (registered trademark). This communication unit 317 can also transmit equirectangular projection image data to an external terminal (device) such as the communication terminal 10X.

[0076] The electronic compass 318 calculates the orientation of the wide-angle camera 9 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) in accordance with Exif (Exchangeable image file format) and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was captured and the data size of the image data.

[0077] The gyro sensor 319 detects changes in angle (roll angle, pitching angle, yaw angle) associated with the movement of the wide-angle shooting device 9. These angle changes are an example of related information (metadata) according to Exif data and are used for image processing such as image correction of captured images.

[0078] The accelerometer 320 detects acceleration in three axes. Based on the detected acceleration, it detects the orientation (angle relative to the direction of gravity) of the wide-angle camera 9. Having both the gyro sensor 319 and the accelerometer 320 improves the accuracy of image correction.

[0079] [About 360° video] Next, the usage of the wide-angle shooting device 9 will be explained using Figure 8. Figure 8 is an illustrative diagram of the use of the wide-angle shooting device 9. As shown in Figure 8, the wide-angle shooting device 9 is used, for example, to capture images of surrounding subjects or scenery while being held by a person. In this case, two hemispherical images can be obtained by capturing images of subjects around the person using the image sensors 303a and 303b shown in Figure 7, respectively.

[0080] Next, using Figures 9 and 10, we will outline the process from the image captured by the wide-angle imaging device 9 to the creation of the equirectangular projection image EC and the 360-degree spherical image CE. Figure 9(a) shows the hemispherical image (front side) captured by the wide-angle imaging device 9, Figure 9(b) shows the hemispherical image (rear side) captured by the wide-angle imaging device 9, and Figure 9(c) shows the image represented by equirectangular projection (hereinafter referred to as the "equistrectangular projection image"). Figure 10(a) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular projection image, and Figure 10(b) shows the 360-degree spherical image.

[0081] As shown in Figure 9(a), the image obtained by the image sensor 303a becomes a curved hemispherical image (front side) by the fisheye lens 302a, which will be described later. Similarly, as shown in Figure 9(b), the image obtained by the image sensor 303b becomes a curved hemispherical image (rear side) by the fisheye lens 302b. The hemispherical image (front side) and the hemispherical image (rear side), which is inverted 180 degrees, are then combined by the wide-angle imaging device 9 to create an equirectangular projection image EC, as shown in Figure 9(c).

[0082] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is superimposed to cover the sphere, as shown in Figure 10(a), creating a full-sphere image CE as shown in Figure 10(b). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. Note that the full-sphere image CE may be a still image or a video.

[0083] As described above, the 360-degree spherical image CE is an image pasted to cover a sphere, which can cause discomfort to the human eye. Therefore, by displaying a predetermined area of ​​the 360-degree spherical image CE (hereinafter referred to as the "predetermined area image") as a flat image with less curvature, it is possible to display it in a way that does not cause discomfort to the human eye. This will be explained using Figures 11 and 12. The predetermined area image may be a video or a still image.

[0084] Figure 11 shows the positions of the virtual camera and a predetermined region when the 360-degree image is treated as a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of the operator viewing the 360-degree image CE, which is displayed as a three-dimensional sphere. Figure 12(a) is a stereoscopic perspective view of Figure 11, and Figure 12(b) shows the predetermined region image as displayed on a screen. In Figure 12(a), the 360-degree image shown in Figure 11 is represented as a three-dimensional sphere CS. If the generated 360-degree image CE is a sphere CS, then, as shown in Figure 12, the virtual camera IC is located inside the 360-degree image CE. The predetermined region T in the 360-degree image CE is the imaging area of ​​the virtual camera IC and is identified by predetermined region information indicating the imaging direction and field of view of the virtual camera IC in the three-dimensional virtual space including the 360-degree image CE. The zoom of the predetermined region T can also be represented by moving the virtual camera IC closer to or further away from the 360-degree image CE. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. Therefore, the predetermined region T can be determined by the field of view α and the distance f from the virtual camera IC to the 360-degree spherical image CE (see Figure 13).

[0085] The predetermined region image Q shown in Figure 12(a) is displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in Figure 12(b). The image shown in Figure 12(b) is a predetermined region image represented by the predetermined region information that has been initially set (default). In the following explanation, we will use the imaging direction (ea, aa) and field of view (α) of the virtual camera IC. Note that the predetermined region T may be represented not by the field of view α and distance f, but by the imaging area (X, Y, Z) of the virtual camera IC which is the predetermined region T.

[0086] Furthermore, when the virtual viewpoint of the virtual camera IC moves (also called "changed") to the right (left side in the drawing) from the state shown in Figure 12(a) to the state shown in Figure 12(c), a predetermined region T in the 360-degree image CE moves to a predetermined region T' accordingly, and the predetermined region image Q displayed on the predetermined display changes to a predetermined region image Q'. As a result, the image shown in Figure 12(b) is changed and displayed on the display of the communication terminal 10 to the image shown in Figure 12(d).

[0087] Next, we will explain the relationship between the predetermined region information and the image of the predetermined region T using Figure 13. Figure 13 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 13, "ea" is the elevation angle, "aa" is the azimuth angle, and "α" is the field of view (Angle). That is, the orientation of the virtual camera IC is changed so that the point of fixation of the virtual camera IC, indicated by the imaging direction (ea, aa), becomes the center point CP of the predetermined region T, which is the imaging area of ​​the virtual camera IC. As shown in Figure 13, the center point CP when the diagonal field of view of the predetermined region T, represented by the field of view α of the virtual camera IC, is α becomes the (x, y) parameter of the predetermined region information. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined region T and the center point CP (2L is the diagonal). And, in Figure 13, the trigonometric function shown in (Equation 1) below generally holds true. (L / f) = tan(α / 2) ... (Equation 1) [Functions of the communication system] Next, the functions of the communication system will be explained using Figures 14 and 15. Figures 14 and 15 are functional block diagrams of each device or system that makes up the communication system.

[0088] <Functional configuration of communication terminal 10α (A, B, C)> The communication terminal 10α is a communication terminal 10 that, for example, receives operation instructions for controlling equipment such as a mobile device 20, but does not have a control function for equipment such as a mobile device 20. The communication terminal 10α has a transmitting / receiving unit 11, an operation input receiving unit 12, a communication control unit 13, a narrow-angle shooting unit 14, an audio input unit 15a, an audio output unit 15b, a display control unit 16, and a storage / reading unit 17. Each of these units is a function or means realized by any of the components shown in Figure 4 operating according to instructions from the CPU 101 that follow a program for the communication terminal 10α deployed from the flash memory 104 onto the RAM 103. The communication terminal 10α also has a storage unit 1000 realized by the RAM 103 and the flash memory 104 shown in Figure 4.

[0089] <Functional Configuration of Communication Terminal 10X> The communication terminal 10X is a communication terminal mounted on the mobile device 20. The communication terminal 10X is an example of a communication terminal that has control functions for equipment such as the mobile device 20. In addition to the functional configuration of the communication terminal 10A described above, the communication terminal 10X has an operation rights confirmation unit 18, an operation instruction receiving unit 19a, a 360-degree video receiving unit 19b, an equipment control unit 19c, and an inter-equipment communication unit 19d, etc.

[0090] (Functional configuration of communication terminal 10) Next, the functional configurations of the communication terminals 10 (communication terminals 10α and 10X) will be described in detail. The transmitting / receiving unit 11 transmits and receives various data (or information) with other communication terminals, devices, or systems via the communication network 2. The transmitting / receiving unit 11 starts receiving status information from the communication management system 50 indicating the status of each communication terminal as a candidate destination even before initiating a call with the desired destination terminal. This status information indicates not only the operating status of each communication terminal 10 (whether it is ON or OFF), but also, in the case of an ON terminal, detailed status such as whether it is available for a call or is currently in a call.

[0091] The operation input reception unit 12 receives various inputs from the operator to the communication terminal. For example, when the operator performs an operation to turn on the power of the communication terminal 10, the operation input reception unit 12 receives the operation and controls the power to turn on.

[0092] The communication control unit 13, for example, upon receiving the power-on notification, automatically sends login request information indicating a login request, and the current IP address of the requesting terminal, to the communication management system 50 via the communication network 2 from the transmitting / receiving unit 11. Furthermore, when the operator performs the operation to turn off the power of the communication terminal 10, the transmitting / receiving unit 11 sends status information indicating the power has been turned off to the communication management system 50, after which the operation input reception unit 12 turns off the power. This allows the communication management system 50 to understand that the communication terminal 10 has changed from power-on to power-off. The communication control unit 13 also performs various communication controls, such as establishing and disconnecting sessions for sending and receiving content data with other communication terminals 10 via the relay device 30. In this embodiment, the communication control unit 13 includes the communication ID (Identification) of the communication terminal 10 in the session control information (for example, start request information, start response information, etc., described later) sent to the communication management system 50.

[0093] The communication ID is an example of identification information for an account that can participate in a session that sends and receives content data using the communication terminal 10. The communication ID may be, for example, a user ID which is the identification information of the operator, an application ID which is the identification information of the application, or a contract ID which is the identification information of the subscriber of the communication terminal 10. In addition, the communication ID may be information that is a combination of at least two of the following: letters, numbers, symbols, and various marks. An email address may also be used.

[0094] The narrow-angle shooting unit 14 has the same functions as cameras installed in general digital cameras, smartphones, tablet devices, notebook PCs, etc., and has a field of view of up to 180 degrees. In this respect, it differs from the wide-angle shooting device 9, which has a field of view of 360°. The narrow-angle shooting unit 14 captures images of subjects, etc., and converts the captured data obtained into predetermined image (video) data for output. There is one narrow-angle shooting unit 14 on the front side of the communication terminal (capturing the front image) and one on the back side (capturing the rear image). There may be more shooting units than this.

[0095] The audio input unit 15a converts the operator's voice into an audio signal via the microphone 114, then converts this audio signal into predetermined audio data and outputs it. The audio output unit 15b converts the audio data into an audio signal and outputs it to the speaker, causing the speaker 115 to output sound.

[0096] The display control unit 16, for example, displays image data included in the content data received by the communication terminal 10 on the display unit 109 or the display input unit 110. The display control unit 16 can also transmit destination list information received from the communication management system 50 to the display unit 109, thereby displaying the destination list on the display unit 109.

[0097] The storage / reading unit 17 performs processing such as storing various data in the storage unit 1000 and reading various data stored in the storage unit 1000.

[0098] The storage unit 1000 stores, for example, authentication information such as the aforementioned communication ID and the password corresponding to the communication ID. The storage unit 1000 stores image data and audio data received when making a call with the destination terminal, overwriting the existing data each time it is received. Of these, the image data before overwriting is used to display an image on the display unit 109, etc., and the audio data before overwriting is used to output audio from the speaker 115.

[0099] Next, we will explain the various functional configurations included in the communication terminal 10X.

[0100] The 360-degree video receiver 19b receives equirectangular projection images from the wide-angle shooting device 9 via wireless communication such as Bluetooth® or wired communication such as a USB cable. The equirectangular projection images are videos that are repeatedly transmitted at a frequency that can be considered as a video. However, they may also be still images, or it may be possible to switch between videos and still images.

[0101] The operation instruction receiving unit 19a receives operation information from the communication terminal 10A via the transmitting / receiving unit 11, requesting control of a device (mobile device 20) to the communication terminal 10X. This operation information includes, for example, the communication ID of the communication terminal 10 that sent the operation information, and operation instructions indicating the requested control content.

[0102] The device control unit 19c controls the mobile device 20 based on the operation information received by the operation instruction receiving unit 19a. The operation information is information indicating the instructions for remotely controlling each function of the robot R from the communication terminal α. For example, the operation information may include instructions such as driving the mobile device 20 of the robot R to rotate the robot R to the right, or taking a picture with the wide-angle shooting device 9.

[0103] The inter-device communication unit 19d communicates with the mobile device 20 and the wide-angle shooting device 9 using the short-range wireless communication unit 118.

[0104] When the communication terminal 10X receives operation information transmitted by the communication terminal α, the operation rights for the target operation indicated by this operation information are set for the transmitting communication terminal α (or user).

[0105] The memory unit 1000 stores the operation rights management DB 1006. The operation rights information managed by this operation rights management DB 1006 is the same as the information managed by the communication management system 50 in the operation rights management DB 5006, and is transmitted to the communication terminal 10X each time the communication management system 50 updates the operation rights information. This operation rights information will be explained in detail later. Note that if the communication terminal 10X does not have an operation rights confirmation unit 18, the communication terminal 10X does not need to have the operation rights management DB 1006.

[0106] <Functional configuration of the mobile device> The mobile device 20 includes, for example, an inter-device communication unit 21 and a driving control unit 22. The inter-device communication unit 21 is implemented, for example, by commands from the CPU 401 shown in Figure 6, and an external device I / F 404, or a short-range wireless communication unit 405. Here, the inter-device communication unit 21 communicates with the communication terminal 10X using the short-range wireless communication unit 405.

[0107] The driving control unit 22 controls the movement (driving) of the mobile device 20, such as moving forward, backward, turning left, and turning right, by controlling, for example, the wheel drive unit 406 and the steering unit 407 shown in Figure 6, according to the control content obtained from the communication terminal 10A.

[0108] <Functional Configuration of Wide-Angle Photography Device> The wide-angle imaging device 9 includes, for example, a wide-angle image transmission unit 31, a wide-angle image capturing unit 32, and a viewpoint information management unit 33. The wide-angle image capturing unit 32 captures a wide-angle equirectangular projection image of the surrounding 360 degrees at a predetermined frame rate. Still images may also be captured. The wide-angle image transmission unit 31 is implemented by the communication unit 317 shown in Figure 7, and transmits the equirectangular projection image of the moving or still image to the communication terminal 10X.

[0109] The viewpoint information management unit 33 manages viewpoint information related to the viewpoint of the virtual camera IC shown in Figure 12. The viewpoint information is information for identifying a predetermined region T. The viewpoint information is transmitted to the communication terminal 10X along with the equirectangular projection image. By transmitting the viewpoint information along with the equirectangular projection image to each communication terminal α, each communication terminal 10α can display a general image as shown in Figures 12(b) and (d), according to the instructions (viewpoint information) from the communication terminal 10X, rather than a curved image as shown in Figure 9(c).

[0110] <Functional Configuration of Communication Management System> The communication management system 50 includes a transmitting / receiving unit 51, a terminal authentication unit 52, a terminal management unit 53, a terminal extraction unit 54, a session management unit 55, and a storage / reading unit 57, an operation rights confirmation unit 58, and an operation rights information creation unit 60, etc. Each of these units is a function or means of functioning that is realized by any of the components shown in Figure 5 operating according to instructions from the CPU 501 that follow a program for the communication management system 50 deployed from the HD 504 onto the RAM 503. The communication management system 50 also has a storage unit 5000 realized by the HD 504 etc shown in Figure 5. First, we will explain each management DB stored in the storage unit 5000 of the communication management system 50.

[0111] (Authentication Management DB) [Table 1]

[0112] The storage unit 5000 of the communication management system 50 has an authentication management DB 5002 constructed in it, which consists of an authentication management table 602 as shown in Table 1. In this authentication management table 602, the communication ID of the communication terminal 10 managed by the communication management system 50 is associated with the password corresponding to the communication ID. For example, in the authentication management table 602 shown in Table 1, it is shown that the password for the communication terminal 10 with communication ID "01aa" is "aaaa".

[0113] (Terminal management database) [Table 2]

[0114] The storage unit 5000 of the communication management system 50 has a terminal management DB 5003 constructed in it, which consists of terminal management tables 603 as shown in Table 2. In this terminal management table 603, for each communication ID of a communication terminal 10, the destination name when the communication terminal 10 is the destination, the operating status of each communication terminal 10, the date and time when the login request information described later was received by the communication management system 50, and the IP address of the communication terminal 10 are associated and managed. For example, in the terminal management table 603 shown in Table 2, the communication terminal 10 with communication ID "01aa" is shown to have the terminal name "Japan Head Office" and an operating status of "ON line (communication possible)". Also, the communication terminal 10 with communication ID "01aa" is shown to have the date and time when the login request information was received by the communication management system 50 at "13:40 on April 10, 20xx" and its IP address is "1.2.1.3".

[0115] (Destination list management database) [Table 3]

[0116] The storage unit 5000 of the communication management system 50 has a destination list management DB 5004 constructed in the destination list management table 701 as shown in Table 3. In this destination list management table 701, all the communication IDs of destination terminals registered as candidate destination terminals are associated and managed with the communication ID of the requesting terminal that requests the start of communication in a remote conference. For example, in the destination list management table 701 shown in Table 3, the candidate destination terminals that can request the start of communication from a requesting terminal with communication ID "01aa" are the communication terminals with communication IDs "01b1", "01b2", and "01b3". These candidate destination terminals are updated by the communication management system 50 when an add or delete request is made to the communication management system 50 from any requesting terminal.

[0117] With this configuration, the requesting terminal (e.g., 01aa) can only initiate communication with pre-registered candidate destination terminals (e.g., 01b1). This destination terminal (e.g., 01b1) also cannot communicate with the requesting terminal (e.g., 01aa) unless it has registered the requesting terminal (e.g., 01aa) as a destination terminal in the destination list management table 701. This mechanism is preferable because it reduces the possibility of unexpected communication between communication terminals 10. However, it is also possible to allow communication between communication terminals 10 arbitrarily without requiring registration in the destination list management table 701.

[0118] (Session management database) [Table 4]

[0119] The memory unit 5000 of the communication management system 50 has a session management DB 5005 constructed in it, which consists of a session management table 702 as shown in Table 4. This session management table 702 manages information such as the relay device ID of the relay device 30 used for relaying, the communication ID of the requesting terminal, the communication ID of the destination terminal, and the date and time the session joined, for each session ID, which is the identification information of the session. For example, in the session management table 702 shown in Table 4, it is shown that the session with session ID "se2" is taking place between the requesting terminal with communication ID "01ad" and the destination terminal with communication ID "01ca". It is also shown that the session with session ID "se2" was started on "20xx / 4 / 10 13:11:11" via the relay device 30 with relay device ID "111b".

[0120] [Table 5]

[0121] The storage unit 5000 of the communication management system 50 has an operation rights management DB 5006 constructed, which consists of an operation rights management table 703 as shown in Table 5. In this operation rights management table 703, for each operation rights target (user's operation target) ID, information is associated and managed, including the name of the operation rights target (i.e., the operation target), the communication ID of the terminal (or user) to which the operation rights are set, and information indicating the elapsed time since the application of the operation rights.

[0122] The scope of application of the right of operation includes the objects that can be operated by the user, as well as each function within robot R and each device that constitutes robot R. In other words, from the perspective of the right of operation, it can be described as the scope of application, and from the perspective of operation by the user, it can be described as the objects that can be operated.

[0123] The target ID for operation rights is an example of target identification information used to identify the target to which operation rights apply. The target ID for operation rights is an example of target identification information used to identify the target to which a user remotely operates.

[0124] The terminal (user) to which the operating rights have been set is identification information used to identify the specific communication terminal 10 or specific user to which the operating rights have been set. The elapsed time of application of the operating rights indicates the time elapsed since the operating rights were set.

[0125] (Functional configuration of each function in the communication management system) Next, the functional configuration of the communication management system 50 will be described in detail. The transmitting / receiving unit 51 transmits and receives various data (or information) with other communication terminals, devices, or systems via the communication network 2.

[0126] The terminal authentication unit authenticates the communication terminal 10 by determining whether the combination of communication ID and password included in the login request information received via the transmitting / receiving unit 51 is included in the authentication management DB (Database) 5002.

[0127] The terminal management unit 53 stores and manages the destination name, operating status, date and time of reception of request information, and the IP address of the requesting terminal, etc., associated with each communication ID in the terminal management DB 5003. For example, when an operator turns the power of the communication terminal 10 from ON to OFF, the terminal management unit 53 changes the operating status of the terminal management DB 5003 from ON to OFF based on the status information sent from the communication terminal 10 indicating that the power has been turned OFF.

[0128] The terminal extraction unit 54 searches the destination list management DB 5004 using the communication ID of the requesting terminal that made the login request as the key, and extracts the communication IDs of destination terminals that can communicate with the requesting terminal. Furthermore, the terminal extraction unit 54 also searches the destination list management DB 5004 and extracts the communication IDs of other communication terminals that have registered the above requesting terminal's communication ID as a candidate destination terminal.

[0129] Furthermore, the terminal extraction unit 54 searches the aforementioned terminal management DB 5003 using the communication ID of the extracted candidate destination terminal as a search key, and reads the operational status for each extracted communication ID. This allows the terminal management unit 53 to obtain the operational status of candidate destination terminals that can communicate with the requesting terminal that made the login request. The terminal management unit 53 also searches the aforementioned terminal management DB 5003 using the requesting terminal's communication ID as a search key, and obtains the operational status of the requesting terminal that made the login request.

[0130] The session management unit 55 controls the sessions managed by the communication management system 50. Session control includes, for example, control for establishing a session, control for allowing the communication terminal 10 to join an established session, control for disconnecting a session, and generation of a session ID. The session management unit 55 also stores and manages the communication ID of the communication terminal 10 that requested the session to start, and the communication ID of the destination terminal, etc., in the session management DB 5005, associating them with the session ID, which is the session identification information.

[0131] The storage and reading unit 57 is implemented by instructions from the CPU 501 and the HDD controller 505 shown in Figure 5, or by instructions from the CPU 501. The storage and reading unit 57 stores various data in the storage unit 5000 and reads various data from the storage unit 5000.

[0132] When the communication terminal 10X receives operation information transmitted by the communication terminal α, the operation rights for the target of the operation indicated by this operation information are set for the transmitting communication terminal α (or user).

[0133] The operation rights information creation unit 60 creates operation rights information based on the destination name managed in the terminal management DB 5003 and the target ID, target, and elapsed time of application of the operation rights managed in the operation rights management DB 5006 (see S908).

[0134] [Processing flow of the communication system] Next, we will explain the processing flow of communication system 1.

[0135] <Preparation phase procedures> Figure 16 is a sequence diagram showing an example of the processing during the preparation phase of a communication system. Here, as an example, we will describe the processing during the preparation phase before a session is started between communication terminal 10A and communication terminal 10X. In the following description, it will be assumed that the communication ID of communication terminal 10A is "01aa" and the communication ID of communication terminal 10X is "01b".

[0136] First, when user a, who is the operator of the requesting terminal, communication terminal 10A, performs an operation to turn on the power of communication terminal 10A, for example, the operation input reception unit 12 receives the operation to turn on the power and turns on the power of communication terminal 10A (step S21).

[0137] Then, triggered by the power being turned ON, the communication control unit 13 transmits login request information from the transmitting / receiving unit 11 to the communication management system 50 via the communication network 2 (step S22). Note that the transmission of login request information by turning ON the power of the communication terminal 10A is just one example; it may also be transmitted by, for example, an operation on the input unit 108 by the operator or the launch of an application.

[0138] Furthermore, the login request information includes a communication ID (the communication ID of the requesting terminal) and a password to identify the communication terminal 10A, which is the requesting device. These communication IDs and passwords are, for example, information read from the storage unit 1000 via the storage / reading unit 17. Also, when login request information is sent from the communication terminal 10A to the communication management system 50, the receiving communication management system 50 can determine the IP address of the sending communication terminal 10A.

[0139] Next, the terminal authentication unit 52 of the communication management system 50 searches the aforementioned authentication management table 602 using the communication ID and password contained in the login request information received via the transmitting / receiving unit 51 as search keys. The terminal authentication unit 52 performs authentication based on whether or not the combination of communication ID and password contained in the login request information received from the communication terminal 10A is included in the authentication management table 602 (step S23).

[0140] If the terminal authentication unit 52 determines that the login request is from a legitimate communication terminal 10A with authorized usage rights, the terminal management unit 53 changes the operational status of the communication ID "01aa" of the communication terminal 10A recorded in the terminal management table 603 to "ON line (communication possible)". At this time, the terminal management unit 53 updates the reception date and time, and, if necessary, updates the IP address of the communication terminal 10 (step S24). As a result, the terminal management table 603 manages the communication ID "01aa" of the communication terminal 10A in association with the operational status "ON line (communication possible)", the reception date and time "20xx.4.10.13:40", and the IP address of the communication terminal 10A "1.2.1.3".

[0141] Then, the transmitting / receiving unit 51 of the communication management system 50 transmits the authentication result information, which is the authentication result obtained by the terminal authentication unit 52, to the communication terminal 10A, which is the requesting terminal that made the login request, via the communication network 2 (step S25). Here, the case in which the terminal authentication unit 52 determines that the communication terminal has legitimate usage rights will be explained below.

[0142] The terminal extraction unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the requesting terminal (communication terminal 10A) that made the login request as the search key. As a result, the terminal management unit 53 extracts the communication IDs of candidate destination terminals that can communicate with the requesting terminal (communication terminal 10A) (step S26). Here, as an example, it is assumed that "01b1", "01b2", and "01b3" are extracted as the communication IDs of destination terminals corresponding to the communication ID "01aa" of the requesting terminal (communication terminal 10A).

[0143] Next, the terminal extraction unit 54 searches the terminal management table 603 using the communication IDs ("01b1", "01b2", "01b3") of the extracted candidate destination terminals as search keys. By doing so, it reads the operating status for each extracted communication ID and obtains the operating status for each communication ID ("01b1", "01b2", "01b3") (step S27).

[0144] Next, the transmitting / receiving unit 51 transmits destination status information, which includes the operating status of each of the candidate destination terminal communication IDs ("01b1", "01b2", "01b3"), to the requesting terminal (communication terminal 10A) (step S28). This allows the requesting terminal (communication terminal 10A) to understand the current operating status of the candidate destination terminal communication IDs ("01b1", "01b2", "01b3") of the requesting terminal (communication terminal 10A). The communication terminal 10A displays the destination selection screen shown in Figure 16.

[0145] Furthermore, the terminal extraction unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the requesting terminal (communication terminal 10A) that made the login request as the search key. As a result, the terminal extraction unit 54 extracts the communication IDs of other requesting terminals that have registered the communication ID "01aa" of the requesting terminal (communication terminal 10A) as a candidate destination terminal (step S29). In the destination list management table 701 shown in Table 3, the communication IDs of the other requesting terminals that are extracted are "01b1", "01b2", and "01b3".

[0146] Next, the terminal management unit 53 of the communication management system 50 searches the terminal management table 603 using the communication ID "01aa" of the requesting terminal (communication terminal 10A) that made the login request as the search key. As a result, the terminal management unit 53 obtains the operational status of the requesting terminal (communication terminal 10A) that made the login request (step S30).

[0147] Then, the terminal management unit 53 of the communication management system 50 extracts the communication IDs ("01b1", "01b2", "01b3") extracted in step S29 above, and which have an operational status of "ON line (communication possible)" in the terminal management table 603.

[0148] Furthermore, the transmitting / receiving unit 51 transmits destination status information, which includes the communication ID "01aa" and operating status "ON line (communication possible)" of the requesting terminal (communication terminal 10A), to the communication terminal 10X corresponding to the extracted communication ID ("01b1", "01b2", "01b3") (step S31).

[0149] Meanwhile, other communication terminals 10X also perform the same processing as in steps S22 to S32 above, for example, in response to operations such as turning on the power. The power of the communication terminal 10X is turned on by, for example, the administrator of the mobile device 20.

[0150] <Recipient Selection Screen> Figure 17 shows an example of a destination selection screen displayed on the communication terminal 10A. The destination selection screen 1201 shown in Figure 17 displays a message 1202 prompting the operator to select a destination communication terminal 10, and several buttons 1203 for selecting the destination communication terminal 10. The operator of the communication terminal 10A selects a destination terminal to request participation in the session by, for example, selecting one of the displayed buttons 1203. In other words, they select which mobile device 20 to control. As shown in Figure 16, the ability for the operator to select multiple communication terminals 10X allows the operator to understand the situation at different locations from the same location.

[0151] <Communication Processing> Figure 17 is a sequence diagram showing an example of communication processing in communication system 1. Here, an example of a communication management method for initiating communication between communication terminal 10A and communication terminal 10X, which is a device control terminal capable of controlling the mobile device 20, is described.

[0152] The operation input reception unit 12 of the communication terminal 10A accepts the selection operation of the destination terminal (communication terminal 10X) by the operator of the communication terminal 10A (step S901).

[0153] The transmitting / receiving unit 11 of the communication terminal 10A sends start request information to the communication management system 50 requesting the start of a session (step S902). This start request information includes, for example, the communication ID of the requesting terminal, which is the communication ID of the requesting terminal, communication terminal 10A, and the communication ID of the destination terminal, which is the communication ID of the destination terminal, communication terminal 10X. The start request information also includes information such as the IP address of the communication terminal 10A (requesting IP address).

[0154] Upon receiving the start request information from the communication terminal 10A, the terminal management unit 53 of the communication management system 50 updates the terminal management DB 5003 based on the communication ID "01aa" of the requesting terminal (communication terminal 10A) included in the start request information (step S903). For example, the terminal management unit 53 changes the operational status information corresponding to the communication ID "01aa" of the communication terminal 10A to "ON line (communicating)" and also updates the reception date and time information.

[0155] The session management unit 55 of the communication management system 50 sends start request information to the destination terminal, communication terminal 10X, requesting the start of a session (step S904). This start request information includes, for example, the communication ID of the requesting terminal, communication terminal 10A.

[0156] Upon receiving a start request information from the communication management system 50, the communication terminal 10X sends start response information to the communication management system 50 (step S905). This start response information includes, for example, the communication ID of the destination terminal of the communication terminal 10X. In this embodiment, it is assumed that the start response information is sent without any operation on the communication terminal 10X side, but it may be required that the communication terminal 10X be operated by an administrator.

[0157] Upon receiving the start response information from the communication terminal 10X, the terminal management unit 53 of the communication management system 50 updates the terminal management DB 5003 based on the communication ID "01b1" of the communication terminal 10X included in the start response information (step S906). For example, the terminal management unit 53 changes the operational status information corresponding to the communication ID "01b1" of the communication terminal 10X to "ON line (communicating)" and updates the reception date and time information.

[0158] The session management unit 55 of the communication management system 50 assigns a session ID, which is identification information for identifying a session (step S907). The session management unit 55 also stores the created session ID in the session management DB 5005, associating it with the communication ID of the requesting terminal (communication ID of communication terminal 10A) and the communication ID of the destination terminal (communication ID of communication terminal 10X) (step S907).

[0159] The storage / reading unit 57 searches the operation rights management DB 5006 using the communication ID stored in the session management DB 5005 in step S907 as a search key, and reads each operation rights management information (each record) that includes this communication ID (step S908). The storage / reading unit 57 also searches the terminal management DB 5003 using the communication ID stored in the session management DB 5005 in step S907 as a search key, and reads the corresponding destination name (step S908). Then, the operation rights information creation unit 60 creates operation rights information based on the operation rights management information and destination name (step S908).

[0160] The session management unit 55 of the communication management system 50 transmits session information to the relay device 30 (step S909). This session information includes, for example, information such as the session ID created in step S907. The relay device 30 can retrieve the session information from the session management DB 5005 based on the session ID.

[0161] The session management unit 55 of the communication management system 50 transmits start instruction information to the communication terminal 10A to instruct it to start a session, and the operation rights information created in step S908 (step S910a). Similarly, the session management unit 55 of the communication management system 50 transmits start instruction information to the communication terminal 10X to instruct it to start a session, and the operation rights information created in step S908 (step S910b). The start instruction information includes a session ID, and the communication terminal 10 can obtain session information from the session management DB 5005 based on the session ID. As a result, the display control unit 16 of the communication terminal 10A causes the display unit 109 to display the operation rights information screen 900 as shown in Figure 19.

[0162] The control rights information screen 900 displays the current control rights settings. For each target to which the control rights apply (the user's target), the terminal to which the control rights are set (the user to which the control rights are set) and the elapsed time since the control rights were applied are shown. This screen 900 also serves as the screen for starting video streaming, and displays a "Start" button 910 to start video streaming and a "Stop" button 920 to stop starting video streaming. Here, we will continue the explanation assuming that user a presses the "Disclose" button 910.

[0163] When the operation input reception unit 12 receives confirmation that the "Disclose" button 910 has been pressed, the transmitting / receiving unit 11 of the communication terminal 10A establishes a video and audio session with the relay device 30 (step S911a). Alternatively, the transmitting / receiving unit 11 of the communication terminal 10A may establish a session with the relay device 30 upon receiving start instruction information, without requiring the "Disclose" button 910 to be pressed. In this case, the operation rights information shown in Figure 19 is displayed immediately after video distribution begins or during video distribution, based on a predetermined operation. Similarly, the transmitting / receiving unit 11 of the communication terminal 10X establishes a session with the relay device 30 based on the received start instruction information (step 911b). This allows communication terminals 10A and 10B1 to participate in remote communication (video distribution) using the same session. Communication terminals 10B and 10C can also participate in remote communication (video distribution) using the same session through the same process as communication terminal 10A.

[0164] The wide-angle image transmission unit 31 of the wide-angle imaging device 9 transmits the equirectangular projection image to the communication terminal 10X (step S911c). For example, when the communication terminal 10X establishes a session, the communication terminal 10X instructs the wide-angle imaging device 9 to start capturing and transmitting the equirectangular projection image. Alternatively, the wide-angle imaging device 9 may continuously transmit the equirectangular projection image to the communication terminal 10X while the power supply of the wide-angle imaging device 9 is ON. Furthermore, the equirectangular projection image may be transmitted even before the session is established.

[0165] Communication terminals 10A and 10X can participate in a session with the same session ID and send and receive content data such as image data and audio data, thereby enabling, for example, a remote conference (step S912).

[0166] <Remote control processing> Next, using Figures 20 and 21, we will explain the process by which user a remotely controls robot R from communication terminal 10A during video distribution. While video and audio transmission and reception are performed via relay device 30, remote control is performed via a control session through the communication management system. User a is granted exclusive remote control rights for predetermined control targets of robot R (in this case, wide-angle shooting device 9 and mobile device 20). It is also possible to grant control rights to communication terminal 10A instead of user a, but here we will explain them collectively assuming that user a is granted control rights.

[0167] (Remote control (Part 1)) First, we will explain the process (part 1) of remotely controlling a telepresence robot using Figure 20. Figure 20 is a sequence diagram showing the process (part 1) of remotely controlling a telepresence robot.

[0168] In the communication terminal 10A, when the operation input receiving unit 12 receives remote operation from user a, the transmitting / receiving unit 11 transmits operation information to the communication management system 50 (step S201). This operation information includes user a's communication ID, an operation target ID to identify the target of the remote operation, and the operation content. The operation content indicates things like driving the moving device 20 to make robot R move in a straight line, or starting shooting with the wide-angle shooting device 9. As a result, the transmitting / receiving unit 51 of the communication management system 50 receives the operation information.

[0169] In the communication management system 50, the operation rights confirmation unit 58 checks whether user a, indicated by the communication ID received in step S201, has the right to operate the target object, indicated by the target object ID received in step S201 (step S202). Specifically, the operation rights confirmation unit 58 checks whether the pair of communication ID and target object ID received in step S201 is managed in the operation rights management DB 5006.

[0170] If control rights have been set, the transmitting / receiving unit 51 transfers operation information to the communication terminal 10X of the robot R (step S203). As a result, the transmitting / receiving unit 11 of the communication terminal 10X receives the operation information, and the inter-device communication unit 19d transmits control instructions to the target of operation (wide-angle shooting device 9, mobile device 20) based on the operation content.

[0171] On the other hand, if no operating rights are set, the transmitting / receiving unit 51 sends a notification to the transmitting communication terminal 10A stating that remote operation is refused (step S205). As a result, the transmitting / receiving unit 11 of the communication terminal 10A receives the notification of refusal, and the display control unit 16 of the communication terminal 10A displays on the display unit 109 that remote operation has been refused. Note that the process in step S205 may not be executed, and the communication management system 50 may leave the refusal in place.

[0172] This concludes the explanation of remote control using control rights (Part 1).

[0173] (Remote control (part 2)) Next, we will explain the process of remotely controlling the telepresence robot (part 2) using Figure 21. Figure 21 is a sequence diagram showing the process of remotely controlling the telepresence robot (part 2).

[0174] In the communication terminal 10A, when the operation input receiving unit 12 receives remote operation from user a, the transmitting / receiving unit 11 transmits operation information to the communication management system 50 (step S301). This operation information includes user a's communication ID, an operation target ID to identify the target of the remote operation, and the operation content. The operation content indicates things like driving the moving device 20 to make robot R move in a straight line, or starting shooting with the wide-angle shooting device 9. As a result, the transmitting / receiving unit 51 of the communication management system 50 receives the operation information.

[0175] The transmitting / receiving unit 51 of the communication management system 50 transfers operation information to the communication terminal 10X of the robot R (step S302). As a result, the transmitting / receiving unit 11 of the communication terminal 10X receives the operation information.

[0176] Next, in the communication terminal 10X, the operation rights confirmation unit 18 checks whether user a, indicated by the communication ID received in step S302, has the right to operate the target device, indicated by the target device ID received in step S302 (step S302). Specifically, the operation rights confirmation unit 18 checks whether the pair of communication ID and target device ID received in step S302 is managed in the operation rights management DB 1006.

[0177] If no control rights are set, the transmitting / receiving unit 51 of the communication terminal 10X sends a notification to the communication management system 50 that remote control is refused (step S304). As a result, the transmitting / receiving unit 51 of the communication management system 50 receives the notification of refusal. Furthermore, the transmitting / receiving unit 51 forwards the notification of refusal to the communication terminal 10A (step S305). As a result, the display control unit 16 of the communication terminal 10A displays on the display unit 109 that remote control has been refused. Note that the processes in steps S305 and S306 may not be executed, and the communication terminal 10X may remain in the refused state.

[0178] On the other hand, if operating rights are set in step S303, the processes in steps S304 and S305 are not executed, and the inter-device communication unit 19d of the communication terminal 10X transmits a control instruction to the target of operation (wide-angle shooting device 9, mobile device 20, etc.) based on the operation content.

[0179] This concludes the explanation of remote control using control rights (part 2).

[0180] [Examples of application of this embodiment] Next, an example of the application of this embodiment will be explained using Figures 22 to 25.

[0181] <Online Museum Tour> Figure 22 is an illustrative diagram showing this embodiment applied to an online tour of a museum. Figure 23 is an example of a screen displayed on the communication terminal 10A during an online tour of a museum.

[0182] As shown in Figure 22, robot R is located inside the museum and can move around the museum while taking pictures via remote control.

[0183] Meanwhile, the display unit 109 of the communication terminal 10A displays a display screen 200 as shown in Figure 23, controlled by the display control unit 16 of the communication terminal 10A. This display screen 200 displays a display area 210 that shows video footage from inside the museum sent from the robot R's communication terminal 10X. Furthermore, the display area 210 displays information 220 regarding the operating rights of user a of the communication terminal 10A. This information regarding operating rights includes the objects to which the operating rights apply, such as the mobile device 20, and the elapsed time since the operating rights were applied to these objects. As a result, user a can understand which objects they can exclusively remotely control and the elapsed time since the application of the operating rights, even while video is being streamed.

[0184] Furthermore, the display area 210 displays multiple operation icons 230 for remotely controlling the robot R. User a can remotely control the museum's robot R by performing operations (screen operations) on these operation icons 230. Note that the display positions of the operation rights information 220 and the operation icons 230 do not have to be as shown in Figure 23, and may be located outside the display area 210.

[0185] Furthermore, on the right side of the display screen 200, the video 241 from the local site and the videos 242 and 243 sent from other sites are displayed. For example, videos 241, 242, and 243 are the videos of user a, user b, and user c, respectively. This allows user a to stream video while considering their own and others' facial expressions.

[0186] <Remote Shopping> Figure 24 is an illustrative diagram showing the application of this embodiment to remote shopping. Figure 25 is an example of a screen displayed on the communication terminal 10A during remote shopping. Note that the display format shown in Figure 25 is basically the same as the display format shown in Figure 23, and the parts with the same reference numerals indicate the same content, so their explanations are omitted.

[0187] In remote shopping, as shown in Figure 24, when the robot R approaches the store clerk, the display screen 200 of the display unit 109 of the communication terminal 10A displays the clerk and the products, as shown in Figure 25. Therefore, user a can decide which products to purchase based on the video feed from the clerk. They can also receive advice from other users b and c.

[0188] [Other application examples] In this embodiment, a mobile device 20 that travels on land has been described as an example, but the mobile device 20 may fly in the air like a drone. It may also navigate on the sea or move underwater. It may also move underground, through narrow passages, or underground. When moving on land, it may move on wheels, or it may move with multiple legs such as two, three, or four legs, or it may move with caterpillars (registered trademark).

[0189] Furthermore, in this embodiment, the communication terminal 10 communicates via the relay device 30, but the communication terminal 10 may communicate without going through the relay device. One example of a communication protocol for such communication is WebRTC (Web Real-Time Communication).

[0190] Furthermore, in this embodiment, for the sake of explanation, the communication management system 50 and the relay device 30 have been described as separate devices, but a device that integrates the functions of both may also provide the functions of the communication management system 50 and the relay device 30.

[0191] Furthermore, the communication system 1 may have multiple communication management systems 50, and the functions of the communication management system 50 may be distributed and installed on multiple servers. Also, one or more of the databases that the communication management system 50 has in the storage unit 5000 may reside on the communication network 2.

[0192] [Effects of the Embodiment] As explained above, this embodiment has the effect of suppressing operations performed by multiple users in a chaotic manner, even when multiple users remotely operate a remotely controlled mobile device, by utilizing the right of operation. [Explanation of Symbols]

[0193] 1. Communication System 9. Wide-angle shooting device 10 Communication terminals 10A Communication Terminal 10B Communication terminal (an example of another communication terminal) 10C Communication Terminal (An example of another communication terminal) 10X Communication Terminal 11 Transmitting and Receiving Unit (Example of a transmitting unit, example of a receiving unit) 12. Operation Input Reception Unit (Example of a reception unit) 14 Narrow-angle shooting section 16 Display Control Unit 20 Mobile device 30 Relay device 50 Communication Management Systems 58 Operation Rights Confirmation Unit R Telepresence Robot (An example of a remotely operated mobile device) 109 Display section [Prior art documents] [Patent Documents]

[0194] [Patent Document 1] Japanese Patent Publication No. 2020-17829

Claims

1. A communication terminal that can be remotely controlled for a remotely controlled mobile device, A receiving unit that receives remote control requests for multiple targets of the remotely operated mobile device, A transmitting unit that enables remote control of the multiple target objects by transmitting operation information indicating the remote control of the multiple target objects to which different operating rights apply, A display control unit that displays a screen associated with the communication terminal or the user of the communication terminal to which the current operating rights are set, for each of the multiple operating targets of the remotely operated mobile device, immediately after the start of video distribution or in response to a predetermined operation during video distribution, A communication terminal characterized by having the following features.

2. The communication terminal according to claim 1, characterized in that the display control unit causes the display control unit to display information regarding the operating rights set for another communication terminal or another user having said other communication terminal.

3. The communication terminal according to claim 2, characterized in that the information relating to the operating right displays the elapsed time since the operating right was applied to the target of the operating right.

4. The communication terminal according to any one of claims 1 to 3, characterized in that the remotely operated mobile device is a telepresence robot.

5. A remote control method performed by a remotely controllable communication terminal on a remotely controlled mobile device, The aforementioned communication terminal is The remotely controlled mobile device receives remote control commands for multiple targets, By transmitting operation information indicating the remote operation of the plurality of targets subject to the exclusive operating rights, the plurality of targets are made remotely controllable. For each of the multiple targets of the remotely controlled mobile device, a screen displaying the communication terminal or the user of the communication terminal to which the current operating rights are set will be displayed immediately after video distribution begins or in response to a predetermined operation during video distribution. A remote control method characterized by the following.

6. For a remotely controlled mobile device, and for a remotely controllable communication terminal, The remotely controlled mobile device receives remote control requests for multiple target devices. By transmitting operation information indicating the remote operation of the multiple targets to which the exclusive operating rights apply, the multiple targets are made remotely controllable. For each of the multiple targets of the remotely controlled mobile device, a screen displaying the communication terminal or the user of the communication terminal to which the current operating rights are set will be displayed immediately after video distribution begins or in response to a predetermined operation during video distribution. A program characterized by the following features.

Images