Communication terminal, program, display method, recording medium

Abstract

To provide a communication terminal that makes it easy for an operator to grasp which direction a moving device is facing when the operator is looking in a certain direction of a wide-angle video.SOLUTION: A communication terminal 10A communicates with another communication terminal 10B mounted on a moving device 20 via a network and transmits operation instruction information for the moving device to the other communication terminal. The communication terminal includes: communication means 11 for receiving a wide-angle video wider than a predetermined video captured by a wide-angle imaging device mounted and connected to the moving device or the other communication terminal; and display processing means 16 for displaying, on a display device, a predetermined area video image, which is part of a video reflected in an area of a display device of the communication terminal, of part of the wide-angle video received by the communication means, and information about the orientation of the moving device.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a communication terminal, a program, a display method, and a recording medium. [Background technology]

[0002] Communication systems have become widespread, in which communication terminals communicate between multiple locations via communication networks such as the Internet or LANs (Local Area Networks) to transmit and receive video and audio between them. Using a communication system, users in remote locations can communicate, for example, to hold a video conference.

[0003] Telepresence robots, which are robots that use communication systems, are also known. Telepresence robots are robots that combine "video conferencing" and "remote control technology." By combining remote control and robotics technology, an operator can operate a robot from a distance and make it behave as if the operator is actually present in a certain location.

[0004] Furthermore, a technology has been devised for mounting multiple cameras on a robot (see, for example, Patent Document 1). Patent Document 1 discloses a moving body equipped with a front camera and a rear camera that capture images in front of and behind the traveling device, as well as an overhead camera that can capture the entire surroundings around the robot. Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, the operator can display any direction within a wide-angle image, such as a full-surround image, which creates the problem that it is difficult for the operator to grasp in which direction the mobile device is facing when looking in a certain direction within the wide-angle image.

[0006] For example, if an operator moves the mobile device forward when the direction in which the operator is looking is different from the front direction of the mobile device, the mobile device will move in a direction that the operator cannot see, thereby reducing operability.

[0007] In view of the above-mentioned problems, an object of the present invention is to provide a communication terminal that allows an operator to easily grasp in which direction the mobile device is facing when the operator is looking in a certain direction of a wide-angle image. [Means for solving the problem]

[0008] In view of the above-mentioned problems, the present invention provides a communication terminal that communicates with another communication terminal mounted on a mobile device via a network, the communication terminal including: a communication means for receiving wide-angle image captured by a wide-angle image capture device mounted on the mobile device or the other communication terminal; and a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of the communication terminal that is different from the other communication terminal, the display processing means displaying an image of the mobile device that allows the orientation of the mobile device to be grasped on the display unit on which at least a part of the wide-angle image is displayed; and the image of the mobile device that is displayed in a manner that allows the orientation of the mobile device to be grasped changes when at least a part of the wide-angle image displayed on the display unit changes. change It is characterized by changing direction depending on the [Effects of the Invention]

[0009] A communication terminal can be provided that allows an operator to easily grasp in which direction the mobile device is facing when looking in a certain direction of a wide-angle image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of an outline of a communication system. [Figure 2] FIG. 1 is a diagram illustrating a display example of a spherical image captured by a wide-angle imaging device. [Figure 3] FIG. 1 is a diagram illustrating an example of the configuration of a communication system according to an embodiment. [Figure 4]FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication terminal according to an embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication management system. [Figure 6] FIG. 2 illustrates an example of a hardware configuration of a mobile device. [Figure 7] FIG. 1 is a diagram illustrating a hardware configuration of an example of a wide-angle imaging device. [Figure 8] FIG. 1 is a diagram illustrating an example of an image of a wide-angle imaging device in use. [Figure 9] 1A and 1B are diagrams illustrating an outline of a process for creating an equirectangular projection image and a celestial sphere image from an image captured by a wide-angle imaging device. [Figure 10] 1A and 1B are diagrams illustrating an outline of a process for creating an equirectangular projection image and a celestial sphere image from an image captured by a wide-angle imaging device. [Figure 11] FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. [Figure 12] FIG. 12 is a diagram showing an example of a three-dimensional perspective view of FIG. 11. [Figure 13] 10 is a diagram illustrating an example of a relationship between predetermined area information and an image of a predetermined area. [Figure 14] 2 is a functional block diagram showing a communication management system included in the communication system, a communication terminal 10A, and functions of the communication terminal 10A in blocks. FIG. [Figure 15] FIG. 10 is a sequence diagram illustrating an example of a process in a preparation stage for communication in the communication system. [Figure 16] FIG. 10 is a diagram illustrating an example of a destination selection screen displayed on the communication terminal. [Figure 17] FIG. 10 is a sequence diagram illustrating an example of communication processing in the communication system. [Figure 18] 10A and 10B are diagrams illustrating an example of a reference imaging direction that serves as a reference for an imaging direction and the front direction of a mobile device. [Figure 19] FIG. 10 is a diagram showing an example of the relationship between an arbitrary imaging direction displayed by an operator and a reference imaging direction. [Figure 20] 1 is a diagram illustrating an example of a perspective projection transformation; [Figure 21] FIG. 10 is a diagram showing an example of a spherical video display screen displayed in pattern 1. [Figure 22] 10 is a flowchart illustrating an example of a procedure in which a communication terminal displays a predetermined area image on a display unit. [Figure 23] FIG. 10 is a diagram showing an example of a polygon model of a telepresence robot and an object showing an imaging direction arrow. [Figure 24] 10 is a diagram showing another example of display of the imaging direction arrow in pattern 2. FIG. [Figure 25] 10 is a flowchart illustrating an example of a procedure in which a communication terminal displays a spherical video on a display unit in pattern 2. FIG. [Figure 26] 10 is a flowchart illustrating an example of a procedure in which a communication terminal displays a spherical video on a display unit in Pattern 3. FIG. [Figure 27] This is an example of a simultaneous video display screen that simultaneously displays a spherical video and a front video (or a rear video). DETAILED DESCRIPTION OF THE INVENTION

[0011] A communication device and a display method performed by the communication device will be described below as an example of an embodiment of the present invention.

[0012] <Outline of operation of communication system and communication terminal> First, an overview of the communication system will be explained using Fig. 1. Fig. 1(a) shows an example of a schematic configuration diagram of a communication system using a telepresence robot. A telepresence robot is placed in a company 801, and an operator 810 is present at home 802. The telepresence robot has a communication terminal 10B and a wide-angle imaging device 9 mounted on a mobile device 20. The operator 810 also has a communication terminal 10A.

[0013] Wide-angle imaging device 9 transmits a celestial sphere image (an example of a second image) to communication terminal 10B, which will be described later. Communication terminals 10A and 10B each have a planar imaging device and a microphone, and transmit and receive images and audio to and from each other. That is, communication terminal 10B transmits a celestial sphere image and a front image 831 (or / and a rear image 832) to communication terminal 10A, and communication terminal 10A transmits the front image 831 to communication terminal 10B.

[0014] Furthermore, application software (hereinafter referred to as an app) that accepts operations related to the movement of mobile device 20 is running on communication terminal 10A, and operator 810 inputs operation instructions while checking the video transmitted by communication terminal 10B. The operation instructions input by operator 810 are transmitted from communication terminal 10A to communication terminal 10B.

[0015] The communication terminal 10B and the mobile device 20 can communicate via short-range wireless communication such as Bluetooth (registered trademark), and the communication terminal 10B controls the mobile device 20 based on operation instructions, so that the operator 810 can move the mobile device 20 from a remote location (home).

[0016] By moving the mobile device 20 in this way, as shown in Fig. 1(b), the operator 810 can converse with a desired person 820. In Fig. 1(b), the operator 810 moves the mobile device 20 to the location of person A 820a to converse with person A, and then moves the mobile device 20 to the location of person B 820b to converse with person B.

[0017] In addition to meeting people, the operator 810 can also do many things remotely, such as checking products in a store in a remote location, inspecting a factory, or attending an exhibition, without having to travel to the location.

[0018] Furthermore, if the planar imaging device built into communication terminal 10B can only capture images of the front and rear, it is difficult for operator 810 to check the situation around mobile device 20 (for example, underfoot or to the left and right). Therefore, in this embodiment, communication terminal 10B is equipped with wide-angle imaging device 9 that can capture images of the surroundings 360 degrees. Wide-angle imaging device 9 may be built into communication terminal 10B or may be built into mobile device 20. Since wide-angle imaging device 9 can capture images over a wide angle range, it is disposed, for example, above where the surroundings are less likely to be obstructed. In FIG. 1, wide-angle imaging device 9 is disposed on the upper side of communication terminal 10B, but it may also be disposed on the left or right side, or on the pole of mobile device 20. Wide-angle imaging device 9 may be built into communication terminal 10B or may be externally attached.

[0019] <Example of spherical video display> 2, a display example will be described in which a display is made so that the operator can grasp the direction in which mobile device 20 is facing when the operator is looking in a certain direction of a celestial sphere image that captures a subject in 360 degrees around the operator. Fig. 2 is a diagram illustrating a display example of a predetermined area image that is a part of the celestial sphere image captured by wide-angle imaging device 9. In this embodiment, three patterns of display examples that enable the operator to grasp the direction in which mobile device 20 is facing will be described.

[0020] Since the orientation of the mobile device 20 is the same as that of the telepresence robot, the mobile device 20 and the telepresence robot may be considered to be the same in terms of orientation. The orientation of the mobile device 20 refers to the direction in which the mobile device 20 is facing, for example, the front direction or the forward direction. As long as the operator can grasp the direction in which the mobile device 20 is facing, the orientation of the mobile device 20 may be sideways, upward, downward, etc.

[0021] 2(a) shows an example of a spherical image display screen 1010 that displays a spherical image according to pattern 1. Pattern 1 allows the operator to grasp the orientation of the mobile device 20 by superimposing an image 840 of a telepresence robot on the spherical image that the operator is currently viewing. The operator can see in which direction the operator is looking relative to the orientation of the mobile device 20. Note that the image 840 of the telepresence robot is displayed in a manner that allows the operator to understand the orientation of the telepresence robot.

[0022] The operator causes communication terminal 10A to display any predetermined area image. The image displayed on spherical image display screen 1010 is the predetermined area image. In FIG. 2(a), part of an open living room is displayed. The predetermined area image is an image of a part of the spherical image that is displayed on a display device (display unit 109, described later) of communication terminal 10A. Display unit 109 provides an area, and the user displays the predetermined area image within this area.

[0023] Therefore, it does not necessarily coincide with the front direction of the mobile device 20. The communication terminal 10A rotates the telepresence robot based on the difference between the front direction of the mobile device 20 and the imaging direction that defines the predetermined area image (the direction that indicates the center of the predetermined area image), and displays the rotated telepresence robot in the omnidirectional image. The displayed image 840 of the telepresence robot is not a telepresence robot in real space, but is created using CG (computer graphics). CG is an image drawn using a computer. CG is mainly available in two dimensions and three dimensions, but in this embodiment, three-dimensional CG is used.

[0024] Therefore, it is possible to see in which direction the telepresence robot is facing in the spherical image. In the example of Figure 2(a), the telepresence robot is facing slightly to the right of the operator. By looking at image 840 of the telepresence robot, the operator can easily determine in which direction the robot will move if it continues moving forward.

[0025] 2(a), for convenience of explanation, the entire image of the telepresence robot 840 is displayed, but since the omnidirectional video is positioned above the telepresence robot, it is preferable to display the telepresence robot image 840 closer to the viewpoint (closer). Therefore, the entire image of the telepresence robot 840 does not need to be displayed.

[0026] Next, pattern 2 will be described. Fig. 2(b) shows an example of a spherical image display screen 1010 that displays a spherical image according to pattern 2. In pattern 2, the direction in which the operator is looking relative to the telepresence robot is displayed, separate from a predetermined area image 853 that is part of the spherical image that the operator is looking at. Specifically, an image 854 of the telepresence robot and an imaging direction arrow 8 are displayed.

[0027] As shown in FIG. 2(b), communication terminal 10A displays an arbitrary predetermined area image 853 in a spherical image field 1001 in response to an operation by an operator. Displayed products are shown in FIG. 2(b). Communication terminal 10A also has a telepresence robot display field 1002 separate from spherical image field 1001. Telepresence robot display field 1002 displays an imaging direction arrow 8 indicating the direction in which predetermined area image 853 is viewed from the telepresence robot. That is, imaging direction arrow 8 three-dimensionally represents the imaging direction viewed by the operator. "Three-dimensionally" means that it includes information on azimuth and elevation angles. The displayed image 854 of the telepresence robot is not a telepresence robot in real space, but is created using CG. The same is true for imaging direction arrow 8. In FIG. 2(b), the imaging direction is the front direction of the telepresence robot.

[0028] In this way, the operator can see which direction the spherical image is viewed from the telepresence robot. The operator can also see which direction the telepresence robot is facing. Also, in Figure 2(b), it can be seen that the telepresence robot approaches the product by moving forward. Compared to pattern 1, this has the advantage that the telepresence robot does not overlap with the spherical image.

[0029] FIG. 2(c) shows an example of a celestial sphere image display screen 1010 that displays a celestial sphere image according to pattern 3. The image displayed on the celestial sphere image display screen 1010 is a predetermined area image. In pattern 3, the communication terminal 10A displays the operation button 860 only when the center direction of the predetermined area image (the above-mentioned imaging direction) and the front direction of the telepresence robot approximately match. In the left diagram c-1 of FIG. 2(c), the imaging direction and the front direction of the telepresence robot do not approximately match, so the operation button 860 is not displayed. In the right diagram c-2 of FIG. 2(c), the imaging direction and the front direction of the telepresence robot approximately match, so the operation button 860 is displayed. "Approximate match" means that the match is sufficient to the extent that it does not interfere with the movement of the mobile device 20. If perfect match were required, the operation button 860 would rarely be displayed and would be inoperable, so the match is determined within a range that does not impair operability. For example, a deviation of about ±10 to ±20 degrees in both the horizontal and elevation directions may be permitted. The operator may also be able to set the magnitude of the deviation.

[0030] In pattern 3, operation button 860 is displayed when the imaging direction and the front direction of the telepresence robot approximately match, so the operator can determine that the imaging direction and the front direction of the telepresence robot approximately match when operation button 860 is displayed after changing the predetermined area image. Therefore, the imaging direction and the direction of the omnidirectional image viewed by the operator can be made to match. Furthermore, operation button 860 is displayed only when the front direction of the telepresence robot and the direction of the omnidirectional image viewed by the operator are made to match, so the operator can know where the robot will move if it is moved forward.

[0031] <Terminology> The operator 810 is a person who operates the mobile device 20. The operator 810 is also a user who uses the communication terminal 10A, but is referred to as the operator in this embodiment.

[0032] The conversation partner 820 is a person in the company or the like, but may also be an animal such as a dog or a cat. Furthermore, the mobile device 20 does not have to be used by the operator 810 to talk to the other party. For example, when the mobile device 20 is patrolling inside the company or the like, the communication terminal 10B only needs to transmit video and audio to the communication terminal 10A, and the operator may not be involved in the conversation.

[0033] The location to which the moving device 20 moves may be any location that the moving device 20 can move to. Furthermore, the device is not limited to one that moves using power such as a motor, and the movement may be assisted by a human.

[0034] A frontal image refers to an image with a normal angle of view, while a spherical image refers to an image with a wider angle than a frontal image (wide-angle image). A normal angle of view is an angle of view in which distortion is acceptable even when captured with a perspective projection lens, and can be simply referred to as an image captured with a perspective projection lens. A wide-angle image refers to an image captured with a wide-angle lens such as a fisheye lens. Projection methods include stereoscopic projection, equidistant projection, equisolid angle projection, and orthogonal projection. A wide-angle image is not limited to a spherical image capturing a 360-degree surrounding area, but may also be an image capturing a hemisphere or an image capturing 180 to 360 degrees horizontally.

[0035] In this embodiment, when a moving image is assumed, it is referred to as video, and when a single still image is assumed, it is referred to as image; however, video has multiple images, and no strict distinction is made between video and images.

[0036] <System configuration example> FIG. 3 is a diagram illustrating an example of the configuration of a communication system according to an embodiment. The communication system 1 includes a plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E), a mobile device 20, a relay device 30, and a communication management system 50. In the following description, the term "communication terminal 10" will be used to refer to any one of the plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E). The term "communication terminal 10B" will be used to refer to any one of the communication terminals (10B1, 10B2, 10B3). The number of communication terminals 10 in FIG. 3 is merely an example.

[0037] The communication terminal 10, the relay device 30, and the communication management system 50 are each connected to be able to communicate with other communication terminals 10, devices, and systems via a communication network 2. The communication network 2 may include, for example, a LAN (Local Area Network), the Internet, a mobile phone network, or a dedicated line.

[0038] The communication terminal 10 is, for example, a general-purpose information processing device such as a tablet terminal, a smartphone, or a personal computer (PC), or a communication terminal such as a dedicated video conference device. The communication terminal 10 can, for example, hold a video conference or the like by transmitting and receiving, for example, image data, audio data, etc., between one or more other communication terminals 10.

[0039] By executing an application corresponding to the communication system 1, the communication terminal 10A can hold a video conference with the communication terminal 10B and remotely control the mobile device 20 via the communication terminal 10B. For example, the communication terminal 10A can move the mobile device 20 equipped with the communication terminal 10B, for example, forward and backward, left and right, etc., by operating operation buttons displayed on the display screen of the video conference. The communication terminal 10A can also run browser software, which can display images and accept operations from the mobile device 20 on the browser software.

[0040] The mobile device 20 and the communication terminal 10B (including the wide-angle imaging device 9) are called a telepresence robot. The mobile device 20 is a device with a traveling function that moves in directions such as "forward," "backward," "right turn," and "left turn" by, for example, driving a plurality of wheels in response to control from the communication terminal 10B attached to the mobile device 20. When a user commands movement to the left or right, the mobile device moves forward after "turning right" or "turning left." Note that the appearance of the mobile device 20 shown in FIG. 3 is merely an example. It is sufficient that the mobile device 20 is capable of moving together with the communication terminal 10B in response to operation commands from the communication terminal 10B mounted on the mobile device 20.

[0041] The relay device 30 is, for example, an information processing device or a system including one or more information processing devices, and relays content data (video, audio, operation instruction information) transmitted and received between the plurality of communication terminals 10. However, the plurality of communication terminals 10 may transmit and receive content data directly without going through the relay device 30.

[0042] The communication management system 50 is, for example, an information processing device or a system including one or more information processing devices. The communication management system 50 performs, for example, login authentication from the communication terminal 10, management of the communication status of the communication terminal 10, management of the destination list, and control of sessions in which communication is performed between multiple communication terminals 10 via the relay device 30.

[0043] In one embodiment, a session is realized by relaying content data including image data and sound data (voice and other sounds) between a plurality of communication terminals 10 via the relay device 30.

[0044] In the above configuration, for example, the operator of communication terminal 10A can communicate with communication terminal 10B and remotely move communication terminal 10B and mobile device 20. This allows the operator of communication terminal 10A to move communication terminal 10B and mobile device 20 close to any other party and hold a video conference or the like.

[0045] The communication system 1 also includes a data providing system that transmits content data in one direction from one communication terminal 10 to another communication terminal 10 via a communication management system 50, and a communication system that transmits information, emotions, etc. between multiple communication terminals 10 via the communication management system 50. This communication system is a system for transmitting information, emotions, etc. between multiple communication terminals via a relay device 30, and examples of this communication system include a television (or video) conference system and a videophone system.

[0046] <Hardware configuration> <<Hardware configuration of communication terminal>> 4 is a diagram illustrating an example of the hardware configuration of a communication terminal according to an embodiment of the present invention. Communication terminal 10 includes a typical computer configuration, such as a central processing unit (CPU) 101, a read-only memory (ROM) 102, a random access memory (RAM) 103, a flash memory 104, and a solid-state drive (SSD) 105. Communication terminal 10 also includes a media interface (I / F) 107, an input unit 108, a display unit 109, a network interface (I / F) 111, a camera 112, an image sensor interface (I / F) 113, a microphone 114, a speaker 115, and an audio input / output interface (I / F) 116. Communication terminal 10 also includes an external device connection interface (I / F) 117, a short-range wireless communication unit 118, and a bus 119.

[0047] The CPU 101 is a computing device that reads programs and data from, for example, the ROM 102 or the flash memory 104 and executes processing to realize the various functions of the communication terminal 10. The ROM 102 is a non-volatile memory that pre-stores programs used to start up the CPU 101, such as an IPL (Initial Program Loader). The RAM 103 is a volatile memory that is used as a work area for the CPU 101.

[0048] The flash memory 104 is a storage device that stores, for example, an operating system (OS), application programs, and various types of data. The SSD 105 controls the reading and writing of various types of data from the flash memory 104 under the control of the CPU 101. The media I / F 107 controls the reading and writing (storage) of data from a recording medium 106, which is a recording medium such as a memory card. A computer-readable program that controls the communication terminal 10 can be stored in the recording medium.

[0049] The input unit 108 is an input device for receiving input operations from an operator, such as a touch panel, keyboard, or pointing device. Voice input may also be possible. The display unit 109 is a display device for displaying various information to the operator. Note that the input unit 108 and the display unit 109 may be a display input unit 110 such as a touch panel display in which a touch panel and a display are integrated.

[0050] The network I / F 111 is a communication interface that enables the communication terminal 10 to transmit and receive data via the communication network 2. The camera 112 includes an image sensor for capturing an image of a subject under the control of the CPU 101. The image sensor I / F 113 controls the image capture by the camera 112 and converts the captured data into predetermined image data. The camera 112 is a planar image capture device that captures images with a narrower angle of view than the wide-angle image capture device 9, but has a higher resolution than the wide-angle image capture device 9. Resolution refers to the degree of detail in a digital image, and is expressed by quantifying the fineness of the individual unit points (dots, pixels) that make up a digital image. The unit of resolution is generally "dots." In the case of displays, it is often expressed as a number arranged horizontally and vertically, such as "1024 x 768 dots." When compared over the same imaging range, the number of dots in the wide-angle image capture device 9 is smaller than that of the camera 112.

[0051] The microphone 114 converts the recorded sound into an electrical signal. The speaker 115 converts the audio signal into sound and outputs it. The audio input / output I / F 116 controls the input and output of sound by the microphone 114 and the speaker 115.

[0052] The external device connection I / F 117 is an interface for connecting an external device such as a USB (Universal Serial Bus), etc. The external device includes, for example, the mobile device 20 in FIG.

[0053] The short-range wireless communication unit 118 is a communication interface for communicating with an external device (e.g., the mobile device 20) by short-range wireless communication such as Bluetooth (registered trademark) or Bluetooth (registered trademark) Low Energy. The bus 119 is commonly connected to each of the above components and transmits address signals, data signals, various control signals, etc.

[0054] <<Communication management system, relay device hardware configuration>> 5 is a diagram showing an example of the hardware configuration of a communication management system according to an embodiment. The communication management system 50 has the configuration of a typical computer, and includes, for example, a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD (Hard Disk Drive) 505, a media drive 507, and a display 508. The communication management system 50 also includes a network I / F 509, a keyboard 511, a mouse 512, a CD-ROM drive 514, and a bus 510.

[0055] The CPU 501 is a computing device that reads out programs and data stored in, for example, the ROM 502, the HD 504, etc., and executes processing to realize each function of the communication management system 50. The ROM 502 is a non-volatile memory that pre-stores programs, such as IPL, used to start the CPU 501. The RAM 503 is a volatile memory used as a work area for the CPU 501, etc.

[0056] The HD 504 is a storage device that stores, for example, an OS, programs such as application programs, and various types of data. The HDD 505 controls the reading and writing of various types of data from and to the HD 504 under the control of the CPU 501. The display 508 is a display device that displays, for example, various types of information such as a cursor, menus, windows, characters, or images.

[0057] The network I / F 509 is a communication interface for performing data communication using the communication network 2. The keyboard 511 is an example of an input device for accepting input operations such as characters, numbers, and various instructions from the system administrator. The mouse 512 is an example of a pointing device for accepting operations such as selecting and executing various instructions, selecting a processing target, and moving a cursor from the system administrator.

[0058] The communication management system 50 or the relay device 30 does not normally need to have the display 508, keyboard 511, and mouse 512. In this case, they may be connected as needed.

[0059] A media drive 507 controls reading and writing (storing) of data from and to a recording medium 506 such as a memory card. A CD-ROM drive 514 controls reading and writing of data from and to a disk 513, which is an example of a removable recording medium. A bus 510 electrically connects the above components and transmits address signals, data signals, various control signals, and the like.

[0060] The above computer hardware configuration is merely an example.

[0061] The relay device 30 has the same hardware configuration as the communication management system 50. The programs for the communication terminal 10, the relay device 30, and the communication management system 50 may be recorded on a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc, and a USB memory. Recording media such as a CD-ROM on which each program is stored, as well as the HD 504 on which these programs are stored, may be provided domestically or internationally as a program product.

[0062] <<Hardware configuration of mobile device>> 6 is a diagram showing an example of the hardware configuration of a mobile device 20 according to an embodiment. The mobile device 20 includes, for example, a CPU 401, a RAM 402, a ROM 403, an external device I / F 404, a short-range wireless communication unit 405, a wheel drive unit 406, a steering unit 407, and the like.

[0063] The CPU 401 is a computing device that executes programs stored in the ROM 403 or the like to realize the various functions of the mobile device 20. The RAM 402 is a volatile memory used as a work area or the like for the CPU 401. The ROM 403 is a non-volatile memory that stores programs and the like for the mobile device 20. The ROM 403 may be, for example, a rewritable non-volatile memory such as a flash ROM.

[0064] The external device I / F 404 is a wired communication interface for connecting to the external device connection I / F 117 of the communication terminal 10 via a wired connection and for communication.

[0065] The short-range wireless communication unit 405 is, for example, a wireless communication interface for performing wireless communication using the same wireless communication method as the short-range wireless communication unit 118 of the communication terminal 10. The mobile device 20 may be capable of communicating with the communication terminal 10 via, for example, the external device I / F 404 or the short-range wireless communication unit 405.

[0066] The wheel driving unit 406 is an example of a driving device that drives wheels for moving the moving device 20. The wheel driving unit 406 includes, for example, a motor.

[0067] The steering unit 407 is an example of a steering device that steers the moving device 20 that moves using the wheel drive unit 406. The steering unit 407 may, for example, be a device that changes the direction or inclination of the wheels, or a device that changes the direction of the moving device 20 by controlling the number of rotations or speed of the left and right wheels.

[0068] <<Example of hardware configuration for wide-angle imaging device>> The hardware configuration of wide-angle imaging device 9 will be described using Figure 7. Figure 7 is a hardware configuration diagram of an example of wide-angle imaging device 9. In the following, wide-angle imaging device 9 is an omnidirectional (all-around) wide-angle imaging device 9 that uses two imaging elements, but the number of imaging elements may be two or more. In addition, it does not necessarily have to be a device dedicated to omnidirectional imaging; a regular digital camera, smartphone, or the like may be equipped with an omnidirectional imaging unit as a retrofit, thereby having substantially the same functions as wide-angle imaging device 9.

[0069] As shown in FIG. 7, the wide-angle imaging device 9 is also connected to an imaging unit 301, an image processing unit 304, an imaging control unit 305, a microphone 308, a sound processing unit 309, a CPU (Central Processing Unit) 311, a ROM (Read Only Memory) 312, an SRAM (Static Random Access Memory) 313, a DRAM (Dynamic Random Access Memory) 314, an operation unit 315, a network I / F 316, a communication unit 317, an antenna 317a, an electronic compass 318, a gyro sensor 319, and an acceleration sensor 320.

[0070] Of these, imaging unit 301 includes wide-angle lenses (so-called fisheye lenses) 302a and 302b, each with a field angle of 180° or more for forming a hemispherical image, and two imaging elements 303a and 303b provided corresponding to each wide-angle lens. Imaging elements 303a and 303b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts optical images captured by fisheye lenses 302a and 302b into image data in the form of electrical signals and outputs the converted image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters required for the operation of the imaging elements are set.

[0071] The imaging elements 303a and 303b of the imaging unit 301 are each connected to the image processing unit 304 via a parallel I / F bus. Meanwhile, the imaging elements 303a and 303b of the imaging unit 301 are connected via a serial I / F bus (such as an I2C bus) separately from the imaging control unit 305. The image processing unit 304, the imaging control unit 305, and the sound processing unit 309 are connected to a CPU 311 via a bus 310. Furthermore, the bus 310 is also connected to a ROM 312, an SRAM 313, a DRAM 314, an operation unit 315, a network I / F 316, a communication unit 317, an electronic compass 318, and the like.

[0072] The image processing unit 304 takes in the image data output from the image sensors 303a and 303b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create equirectangular projection image data such as that shown in Figure 9(c).

[0073] The imaging control unit 305 generally sets commands and the like in the registers of the imaging elements 303a and 303b using an I2C bus, with the imaging control unit 305 acting as a master device and the imaging elements 303a and 303b acting as slave devices. Necessary commands and the like are received from the CPU 311. The imaging control unit 305 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 303a and 303b and send it to the CPU 311.

[0074] Furthermore, imaging control unit 305 instructs imaging elements 303a and 303b to output image data when the shutter button on operation unit 315 is pressed. Some wide-angle imaging devices 9 have a preview display function or a function for displaying moving images on a display (for example, a display of communication terminal 10B). In this case, image data is output from imaging elements 303a and 303b continuously at a predetermined frame rate (frames / minute).

[0075] As will be described later, the imaging control unit 305 also functions as a synchronization control means that synchronizes the output timing of image data from the imaging elements 303a and 303b in cooperation with the CPU 311. In this embodiment, the wide-angle imaging device 9 is not provided with a display, but a display unit may be provided.

[0076] The microphone 308 converts sound into sound (signal) data. The sound processing unit 309 takes in the sound data output from the microphone 308 via the I / F bus and performs predetermined processing on the sound data.

[0077] The CPU 311 controls the overall operation of the wide-angle imaging device 9 and executes necessary processing. The ROM 312 stores various programs for the CPU 311. The SRAM 313 and DRAM 314 are work memories that store programs executed by the CPU 311, data in the middle of processing, etc. In particular, the DRAM 314 stores image data in the middle of processing by the image processing unit 304 and data of a processed equirectangular projection image.

[0078] The operation unit 315 is a general term for operation buttons such as a shutter button, etc. The operator operates the operation unit 315 to input various imaging modes, imaging conditions, and the like.

[0079] The network I / F 316 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The network I / F 316 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 314 is recorded on an external medium via the network I / F 316, or transmitted to an external terminal (device) such as the communication terminal 10B via the network I / F 316 as needed.

[0080] The communication unit 317 communicates with an external terminal (device) of the communication terminal 10B by short-range wireless communication technology such as Wi-Fi, NFC, or Bluetooth (registered trademark) via an antenna 317a provided in the wide-angle imaging device 9. The communication unit 317 can also transmit data of the equirectangular projection image to an external terminal (device) such as the communication terminal 10B.

[0081] The electronic compass 318 calculates the direction of the wide-angle imaging device 9 from the Earth's magnetism and outputs the direction information. This direction information is an example of related information (metadata) conforming to the Exchangeable Image File Format (Exif), and is used for image processing such as image correction of the captured image. The related information also includes data such as the date and time the image was captured and the data size of the image data.

[0082] The gyro sensor 319 detects changes in angle (roll angle, pitch angle, yaw angle) accompanying the movement of the wide-angle imaging device 9. The changes in angle are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of captured images.

[0083] The acceleration sensor 320 detects acceleration in three axial directions. Based on the detected acceleration, the attitude (angle with respect to the direction of gravity) of the wide-angle imaging device 9 is detected. Having both the gyro sensor 319 and the acceleration sensor 320 improves the accuracy of image correction.

[0084] <About spherical images> Next, a usage situation of wide-angle imaging device 9 will be described with reference to Fig. 8. Fig. 8 is an image diagram of wide-angle imaging device 9 in use. As shown in Fig. 8, wide-angle imaging device 9 is used, for example, by being held by a user and capturing images of subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using imaging element 303a and imaging element 303b shown in Fig. 7, respectively.

[0085] Next, an outline of the processing up to creating an equirectangular projection image EC and a celestial sphere image CE from an image captured by the wide-angle imaging device 9 will be described with reference to Fig. 9 and Fig. 10. Fig. 9(a) is a diagram showing a hemispherical image (front side) captured by the wide-angle imaging device 9, Fig. 9(b) is a diagram showing a hemispherical image (rear side) captured by the wide-angle imaging device 9, and Fig. 9(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as an "equirectangular projection image"). Fig. 10(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 10(b) is a diagram showing a celestial sphere image.

[0086] As shown in Fig. 9(a), the image obtained by the image sensor 303a becomes a hemispherical image (front side) curved by the fisheye lens 302a (described later). Also, as shown in Fig. 9(b), the image obtained by the image sensor 303b becomes a hemispherical image (rear side) curved by the fisheye lens 302b. The hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are then combined by the wide-angle imaging device 9 to create an equirectangular projection image EC as shown in Fig. 9(c).

[0087] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is pasted to cover the spherical surface as shown in FIG. 10(a), and a celestial sphere image CE as shown in FIG. 10(b) is created. In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the celestial sphere image CE may be a still image or a video.

[0088] As described above, the spherical image CE is an image pasted so as to cover the spherical surface, which gives a sense of incongruity to people when they view it. Therefore, by displaying a predetermined area of ​​the spherical image CE (hereinafter, corresponding to the above-mentioned "predetermined area image") as a flat image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to people. This will be described with reference to FIGS. 11 and 12. The predetermined area image may be a moving image or a still image.

[0089] FIG. 11 is a diagram showing the positions of the virtual camera and the predetermined region when the celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of an operator viewing the celestial sphere image CE displayed as a three-dimensional sphere. FIG. 12(a) is a three-dimensional perspective view of FIG. 11, and FIG. 12(b) is a diagram showing an image of the predetermined region when displayed on a display. In FIG. 12(a), the celestial sphere image shown in FIG. 11 is represented by a three-dimensional sphere CS. If the celestial sphere image CE generated in this manner is a 3D sphere CS, then the virtual camera IC is located inside the celestial sphere image CE as shown in FIG. 12. The predetermined region T in the celestial sphere image CE is an imaging region of the virtual camera IC, and is specified by predetermined region information that indicates the imaging direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere image CE. Zooming of the predetermined region T can also be achieved by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined area video Q is an image of a predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE (see FIG. 13).

[0090] The predetermined area image Q shown in FIG. 12(a) is then displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in FIG. 12(b). The image shown in FIG. 12(b) is a predetermined area image represented by the initial (default) predetermined area information. The following description will be given using the imaging direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may be represented by the imaging area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

[0091] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described using FIG. 13. FIG. 13 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 13, "ea" represents the elevation angle, "aa" represents the azimuth angle, and "α" represents the angle of view (Angle). That is, the posture of the virtual camera IC is changed so that the gaze point of the virtual camera IC, indicated by the imaging direction (ea, aa), becomes the center point CP of the predetermined area T, which is the imaging area of ​​the virtual camera IC. As shown in FIG. 13, when the diagonal angle of view of the predetermined area T, represented by the angle of view α of the virtual camera IC, is α, the center point CP becomes the (x, y) parameter of the predetermined area information. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). In FIG. 13, the trigonometric function shown in Equation 1 below generally holds.

[0092] L / f=tan(α / 2) (Equation 1) <About the function> Next, the functions of the communication system will be described with reference to Fig. 14. Fig. 14 is a functional block diagram showing the functions of the communication management system 50, the communication terminal 10B, and the communication terminal 10A that the communication system 1 has.

[0093] <<Functional configuration of communication terminal 10A>> The communication terminal 10A is a communication terminal 10 that accepts operation instructions for controlling devices such as a mobile device 20, but does not have a function for controlling devices such as a mobile device 20. The communication terminal 10A includes a transmitter / receiver 11, an operation input receiver 12, a communication controller 13, an image capture unit 14, an audio input unit 15a, an audio output unit 15b, a display controller 16, and a memory / readout unit 17. Each of these units is a function or means realized by one of the components shown in FIG. 4 operating in response to an instruction from the CPU 101 in accordance with a program for the communication terminal 10 that is loaded from the flash memory 104 onto the RAM 103. The communication terminal 10 also includes a memory unit 18 realized by the RAM 103 shown in FIG. 4 and the flash memory 104 shown in FIG. 4.

[0094] <<Functional Configuration of Communication Terminal 10B>> The communication terminal 10B is a communication terminal 10 mounted on the mobile device 20. The communication terminal 10B is an example of a communication terminal 10 having a function of controlling devices such as the mobile device 20. In addition to the functional configuration of the communication terminal 10A described above, the communication terminal 10B has an operation instruction receiving unit 19a, a celestial sphere video receiving unit 19b, a device control unit 19c, an inter-device communication unit 19d, and the like.

[0095] (Functional configuration of communication terminal 10) Next, each functional configuration of the communication terminal 10 (communication terminal 10A and communication terminal 10B) will be described in detail.

[0096] The transmitting / receiving unit 11 transmits and receives various data (or information) to and from other communication terminals, devices, or systems via the communication network 2. Before starting a call with a desired destination terminal, the transmitting / receiving unit 11 starts receiving status information indicating the status of each communication terminal as a destination candidate from the communication management system 50. Note that this status information not only indicates the operating status of each communication terminal 10 (whether it is online or offline), but also indicates, in the case of an online state, detailed status such as whether a call is possible or in progress.

[0097] The operation input receiving unit 12 receives various inputs to the communication terminal by an operator. For example, when the operator performs an operation to turn on the power of the communication terminal 10, the operation input receiving unit 12 receives the operation and controls the power to be turned on.

[0098] For example, upon receiving the power-on command, the communication control unit 13 automatically transmits login request information indicating a login request and the current IP address of the requesting terminal from the transmitting / receiving unit 11 to the communication management system 50 via the communication network 2. When the operator performs an operation to power off the communication terminal 10, the transmitting / receiving unit 11 transmits status information indicating that the power is to be turned off to the communication management system 50, and then the operation input receiving unit 12 turns the power off. This allows the communication management system 50 to know that the communication terminal 10 has changed from power-on to power-off.

[0099] The communication control unit 13 also performs various communication controls, such as establishing and disconnecting a session for transmitting and receiving content data to and from other communication terminals 10 via the relay device 30. The communication control unit 13 according to this embodiment transmits session control information (for example, start request information, start response information, etc., which will be described later) to the communication management system 50 together with the communication ID (Identification) of the communication terminal 10.

[0100] The communication ID is an example of identification information of an account that can participate in a session for transmitting and receiving content data using the communication terminal 10. The communication ID may be, for example, a user ID that is identification information of an operator, an application ID that is identification information of an application, or a contract ID that is identification information of a subscriber of the communication terminal 10. The communication ID may also be information that combines at least two of letters, numbers, symbols, and various marks. An email address may also be used.

[0101] The imaging unit 14 converts imaging data obtained by imaging a subject into predetermined image (video) data and outputs it. There is one imaging unit 14 on the front side of the communication terminal (capturing a front video image) and one on the back side (capturing a rear video image). There may be more imaging units than these.

[0102] After the operator's voice is converted into a voice signal by the microphone 114, the voice input unit 15a converts the voice signal into predetermined voice data and outputs it. The voice output unit 15b converts the voice data into a voice signal and outputs it to the speaker 115, causing the speaker 115 to output the voice.

[0103] The display control unit 16, for example, displays image data included in content data received by the communication terminal 10 on the display unit 109 or the display input unit 110. The display control unit 16 also transmits information on the destination list received from the communication management system 50 to the display unit 109, and causes the display unit 109 to display the destination list.

[0104] The storage / readout unit 17 stores various data in the storage unit 18 and reads out various data stored in the storage unit 18 .

[0105] The storage unit 18 stores authentication information such as the above-mentioned communication ID and a password corresponding to the communication ID. Image data and audio data received when making a call with a destination terminal are overwritten and stored in the storage unit 18 every time they are received. Of these, images are displayed on the display unit 109 etc. using the image data before being overwritten, and audio is output from the speaker 115 using the audio data before being overwritten.

[0106] Next, each functional configuration included in communication terminal 10B will be described.

[0107] The omnidirectional image receiving unit 19b receives the equirectangular projection image from the wide-angle imaging device 9 via wireless communication such as Bluetooth (registered trademark) or wired communication such as a USB cable. The equirectangular projection image is a moving image that is repeatedly transmitted frequently enough to be considered a moving image. However, the image may be a still image, or may be switchable between a moving image and a still image.

[0108] The operation instruction receiving unit 19a receives operation instruction information requesting communication terminal 10B to control a device (mobile device 20) from communication terminal 10A via the transmitting / receiving unit 11. This operation instruction information includes, for example, the communication ID of communication terminal 10 that has transmitted the operation instruction information, an operation instruction indicating the requested control content, and the like.

[0109] The device control unit 19c controls the mobile device 20 based on the operation instruction included in the operation instruction information received by the operation instruction receiving unit 19a. The inter-device communication unit 19d communicates with the mobile device 20 using the short-range wireless communication unit 118.

[0110] <<Functional configuration of mobile device>> The mobile device 20 includes, for example, an inter-device communication unit 21 and a travel control unit 22. The inter-device communication unit 21 is realized by, for example, instructions from a CPU 401 shown in Fig. 6, an external device I / F 404, or a short-range wireless communication unit 405. Here, it is assumed that the inter-device communication unit 21 communicates with the communication terminal 10B using the short-range wireless communication unit 405.

[0111] The traveling control unit 22 controls the movement (travel) of the moving device 20, such as forward, backward, left turn, right turn, etc., by controlling, for example, the wheel driving unit 406 and the steering unit 407 in FIG. 6 according to the control content acquired from the communication terminal 10A.

[0112] <<Functional configuration of wide-angle imaging device>> Wide-angle imaging device 9 includes, for example, wide-angle image transmission unit 31 and wide-angle image capturing unit 32. Wide-angle image capturing unit 32 captures a 360-degree wide-angle equirectangular projection image at a predetermined frame rate. Still images may also be captured. Wide-angle image transmission unit 31 is realized by communication unit 317 shown in FIG. 7, and transmits a moving or still equirectangular projection image to communication terminal 10B.

[0113] <<Functional configuration of the communication management system>> The communication management system 50 includes a transmission / reception unit 51, a terminal authentication unit 52, a terminal management unit 53, a terminal extraction unit 54, a session management unit 55, and a storage / readout unit 57. Each of these units is a function or a means for performing the function that is realized when any of the components shown in Fig. 5 operates in response to an instruction from the CPU 501 in accordance with a program for the communication management system 50 that is loaded from the HD 504 onto the RAM 503. The communication management system 50 also includes a storage unit 5000 that is realized by the HD 504 etc. shown in Fig. 5.

[0114] (Functional configuration of communication management system) Next, a detailed description will be given of each functional configuration of the communication management system 50. The transmitting / receiving unit 51 transmits and receives various data (or information) to and from other communication terminals, devices, or systems via the communication network 2.

[0115] The terminal authentication unit 52 authenticates the communication terminal 10 by determining whether the combination of communication ID and password included in the login request information received via the transmission / reception unit 51 is included in the authentication management DB (Database) 5002.

[0116] The terminal management unit 53 stores and manages, in association with each other, the destination name, operating status, the date and time of receipt of request information, etc., and the IP address of the requesting terminal for each communication ID in the terminal management DB 5003. For example, when an operator turns the power of the communication terminal 10 from ON to OFF, the terminal management unit 53 changes the operating status in the terminal management DB 5003 from ON to OFF based on the status information sent from the communication terminal 10 indicating that the power is to be turned OFF.

[0117] The terminal extraction unit 54 searches the destination list management DB 5004 using the communication ID of the requesting terminal that has made the login request as a key, and extracts the communication IDs of destination terminals that can make calls to the requesting terminal. Furthermore, the terminal extraction unit 54 searches the destination list management DB 5004 and extracts the communication IDs of other communication terminals that have registered the communication ID of the requesting terminal as a candidate destination terminal.

[0118] Furthermore, the terminal extraction unit 54 searches the above-mentioned terminal management DB 5003 using the communication IDs of the extracted destination terminal candidates as search keys, and reads out the operating status for each extracted communication ID. This allows the terminal management unit 53 to obtain the operating status of destination terminal candidates that can communicate with the request source terminal that has made the login request. The terminal management unit 53 also searches the above-mentioned terminal management DB 5003 using the request source communication ID as a search key, and obtains the operating status of the request source terminal that has made the login request.

[0119] The session management unit 55 controls sessions managed by the communication management system 50. Session control includes, for example, control for establishing a session, control for allowing a communication terminal 10 to participate in an established session, control for disconnecting a session, generation of a session ID, etc. The session management unit 55 also stores and manages the communication ID of the communication terminal 10 that requested the start of a session and the communication ID of the destination terminal, etc., in association with the session ID, which is session identification information, in the session management DB 5005.

[0120] 5 and the HDD 505, or is realized by an instruction from the CPU 501. The storage and reading unit 57 stores various data in the storage unit 5000 and reads various data from the storage unit 5000.

[0121] <Examples of information managed by the communication management system> Each management DB stored in the storage unit 5000 of the communication management system 50 will be described.

[0122] [Table 1] The authentication management DB 5002 stored in the storage unit 5000 of the communication management system 50 includes, for example, an authentication management table 602 as shown in Table 2. In this authentication management table 602, the communication IDs of the communication terminals 10 managed by the communication management system 50 are associated with the passwords corresponding to the communication IDs and are managed. For example, the authentication management table 602 shown in Table 2 indicates that the password for the communication terminal 10 with the communication ID "01aa" is "aaaa."

[0123] [Table 2] The terminal management DB 5003 stored in the storage unit 5000 of the communication management system 50 includes a terminal management table 603 such as that shown in Table 2. In this terminal management table 603, for each communication ID of a communication terminal 10, the destination name when each communication terminal 10 is the destination, the operating status of each communication terminal 10, the date and time when login request information described below was received by the communication management system 50, and the IP address of the communication terminal 10 are associated and managed. For example, the terminal management table 603 shown in Table 3 indicates that the communication terminal 10 with the communication ID "01aa" has the terminal name "Japan Headquarters" and the operating status "ONLINE (communication available)". It also indicates that the date and time when login request information was received by the communication management system 50 for the communication terminal 10 with the communication ID "01aa" was "April 10, 20xx, 1:40 PM" and the IP address is "1.2.1.3".

[0124] [Table 3] The destination list management DB 5004 stored in the storage unit 5000 of the communication management system 50 includes, for example, a destination list management table 701 as shown in Table 3. In this destination list management table 701, the communication IDs of all destination terminals registered as candidate destination terminals are managed in association with the communication ID of a requesting terminal requesting the start of communication in a video conference. For example, in the destination list management table 701 shown in Table 4, the candidate destination terminals to which a requesting terminal with a communication ID of "01aa" can request the start of communication are the communication terminals with communication IDs of "01b1," "01b2," and "01b3." These candidate destination terminals are updated by being added or deleted by the communication management system 50 in response to an addition or deletion request from any requesting terminal to the communication management system 50.

[0125] According to this configuration, a request source terminal (e.g., 01aa) can only initiate communication with a pre-registered candidate destination terminal (e.g., 01b1). This destination terminal (e.g., 01b1) cannot communicate with the request source terminal (e.g., 01aa) unless the request source terminal (e.g., 01aa) is registered as a destination terminal in the destination list management table 701. This mechanism is preferable in that it reduces the possibility of unexpected communication terminals 10 communicating with each other. However, it is also possible to eliminate the need for registration in the destination list management table 701 and allow communication terminals 10 to communicate with each other arbitrarily.

[0126] [Table 4] The session management DB 5005 stored in the storage unit 5000 of the communication management system 50 includes a session management table 702 such as that shown in Table 4. This session management table 702 manages, for each session ID, which is session identification information, information such as the relay device ID of the relay device 30 used for relay, the communication ID of the requesting terminal, the communication ID of the destination terminal, and the date and time of session participation. For example, the session management table 702 shown in Table 5 indicates that a session with session ID "se2" is being carried out between the communication ID "01ad" of the requesting terminal and the communication ID "01ca" of the destination terminal. It also indicates that the session with session ID "se2" was started at "20xx / 4 / 10 13:11:11" via the relay device 30 with relay device ID "111b."

[0127] <Processing flow> Next, the flow of processing in the communication system 1 will be described.

[0128] (Preparatory processing) 15 is a sequence diagram showing an example of processing in a preparation stage for communication in communication system 1. Here, as an example, processing in a preparation stage before starting a session between communication terminal 10A and communication terminal 10B1 will be described. In the following description, it is assumed that the communication ID of communication terminal 10A is "01aa" and the communication ID of communication terminal 10B1 is "01b".

[0129] First, when the operator of the communication terminal 10A, which is the requesting terminal, performs an operation to turn on the power of the communication terminal 10A, for example, the operation input receiving unit 12 receives the operation to turn on the power and turns on the power of the communication terminal 10A (step S21).

[0130] Then, in response to the power-on, the communication control unit 13 transmits login request information requesting a login from the transmitting / receiving unit 11 to the communication management system 50 via the communication network 2 (step S22). Note that the login request information being transmitted by an operation to turn on the power of the communication terminal 10A is just an example, and the login request information may be transmitted by, for example, an operation on the input unit 108 by the operator, the launch of an app, or the like.

[0131] The login request information also includes a communication ID (communication ID of the requesting terminal) and a password for identifying the communication terminal 10A that is the requesting terminal itself. The communication ID and password are, for example, information read from the memory unit 18 via the memory / read unit 17. When the login request information is transmitted from the communication terminal 10A to the communication management system 50, the communication management system 50, which is the receiving side, can know the IP address of the communication terminal 10A, which is the transmitting side.

[0132] Next, the terminal authentication unit 52 of the communication management system 50 searches the above-mentioned authentication management table 602 using the communication ID and password included in the login request information received via the transmission / reception unit 51 as search keys. The terminal authentication unit 52 performs authentication based on whether or not the combination of the communication ID and password included in the login request information received from the communication terminal 10A is included in the authentication management table 602 (step S23).

[0133] If the terminal authentication unit 52 determines that the login request is from the communication terminal 10A that has valid usage authority, the terminal management unit 53 changes the operation status of the communication ID "01aa" of the communication terminal 10A recorded in the terminal management table 603 to "ONLINE (communication available)". At this time, the terminal management unit 53 updates the reception date and time, and also updates the IP address of the communication terminal 10A as necessary (step S24). As a result, the communication ID "01aa" of the communication terminal 10A is managed in the terminal management table 603 in association with the operation status "ONLINE (communication available)", the reception date and time "20xx.4.10.13:40", and the IP address "1.2.1.3" of the communication terminal 10A.

[0134] Then, the transmitting / receiving unit 51 of the communication management system 50 transmits authentication result information indicating the authentication result obtained by the terminal authentication unit 52 to the communication terminal 10A, which is the requesting terminal that made the login request, via the communication network 2 (step S25). Here, a case where the terminal authentication unit 52 determines that the communication terminal has valid usage authority will be described below.

[0135] The terminal extraction unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the request source terminal (communication terminal 10A) that has made the login request as a search key. As a result, the terminal management unit 53 extracts communication IDs of candidate destination terminals that can communicate with the request source terminal (communication terminal 10A) (step S26). Here, as an example, it is assumed that "01b1", "01b2", and "01b3" are extracted as communication IDs of destination terminals that correspond to the communication ID "01aa" of the request source terminal (communication terminal 10A).

[0136] Next, the terminal extraction unit 54 searches the terminal management table 603 using the communication IDs ("01b1", "01b2", "01b3") of the extracted destination terminal candidates as search keys. As a result, the operation status of each of the extracted communication IDs ("01b1", "01b2", "01b3") is acquired by reading out the operation status for each of the communication IDs (step S27).

[0137] Next, the transmitter / receiver 51 transmits destination status information including the operation status of each of the communication IDs ("01b1", "01b2", "01b3") of the destination terminal candidates to the request source terminal (communication terminal 10A) (step S28). This allows the request source terminal (communication terminal 10A) to grasp the current operation status of the communication IDs ("01b1", "01b2", "01b3") that are destination terminal candidates for this request source terminal (communication terminal 10A). The communication terminal 10A displays the destination selection screen shown in FIG.

[0138] Furthermore, the terminal extraction unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the request source terminal (communication terminal 10A) that has made the login request as a search key. As a result, the terminal extraction unit 54 extracts the communication IDs of other request source terminals that have registered the communication ID "01aa" of the request source terminal (communication terminal 10A) as a destination terminal candidate (step S29). In the destination list management table 701 shown in Table 3, the communication IDs of the other extracted request source terminals are "01b1", "01b2", and "01b3".

[0139] Next, the terminal management unit 53 of the communication management system 50 searches the terminal management table 603 using the communication ID "01aa" of the request source terminal (communication terminal 10A) that has made the login request as a search key. As a result, the terminal management unit 53 acquires the operating status of the request source terminal (communication terminal 10A) that has made the login request (step S30).

[0140] Then, the terminal management unit 53 of the communication management system 50 extracts the communication IDs ("01b1", "01b2", "01b3") that have an operating status of "ONLINE (communication possible)" in the terminal management table 603 from the communication IDs ("01b1", "01b2", "01b3") extracted in step S29 above.

[0141] In addition, the transmitter / receiver 51 transmits destination status information including the communication ID "01aa" of the requesting terminal (communication terminal 10A) and the operating status "ONLINE (communication available)" to the communication terminal 10B1 corresponding to the extracted communication ID ("01b1", "01b2", "01b3") (step S31).

[0142] Meanwhile, other communication terminal 10B1 also performs the same processes as steps S22 to S32 in response to, for example, a power-on operation, etc. Communication terminal 10B1 is powered on by, for example, an administrator of mobile device 20.

[0143] <Destination selection screen> Fig. 16 is a diagram showing an example of a destination selection screen displayed on communication terminal 10A. Destination selection screen 1201 shown in Fig. 16 displays a message 1202 prompting the operator to select a destination communication terminal 10, and multiple buttons 1203 for selecting destination communication terminals 10. The operator of communication terminal 10A selects a destination terminal to request participation in a session, for example, by selecting one button 1203 from the multiple displayed buttons 1203. In other words, the operator selects which mobile device 20 to control. As shown in Fig. 16, the operator can select multiple communication terminals 10B, allowing the operator to grasp the status of different locations from the same location.

[0144] <Communication processing> 17 is a sequence diagram showing an example of communication processing in communication system 1. Here, an example of a communication management method for starting communication between communication terminal 10A and communication terminal 10B1, which is a device control terminal capable of controlling mobile device 20, will be described.

[0145] In step S901, operation input receiving unit 12 of communication terminal 10A receives an operation to select a destination terminal (communication terminal 10B1) by the operator of communication terminal 10A.

[0146] The transmitter / receiver 11 of communication terminal 10A transmits start request information requesting the start of a session to communication management system 50 (step S902). This start request information includes, for example, the communication ID of the requesting terminal, which is the communication ID of communication terminal 10A, which is the requesting terminal, and the communication ID of the destination terminal, which is the communication ID of communication terminal 10B1, which is the destination terminal. The start request information also includes information such as the IP address of communication terminal 10A (requesting IP address).

[0147] In step S903, the terminal management unit 53 of the communication management system 50 that has received the start request information from the communication terminal 10A updates the terminal management DB 5003 based on the communication ID "01aa" of the request source terminal (communication terminal 10A) included in the start request information. For example, the terminal management unit 53 changes the operating status information corresponding to the communication ID "01aa" of the communication terminal 10A to "ONLINE (communicating)" and also updates the information on the reception date and time.

[0148] In step S904, session management unit 55 of communication management system 50 transmits start request information requesting the start of a session to communication terminal 10B1, which is the destination terminal. This start request information includes, for example, the communication ID of communication terminal 10A, which is the request source terminal.

[0149] In step S905, communication terminal 10B1, which has received the start request information from communication management system 50, transmits start response information to communication management system 50. This start response information includes, for example, the communication ID of the destination terminal of communication terminal 10B1. In this embodiment, the start response information is transmitted without requiring any operation on the side of communication terminal 10B1, but it may also be transmitted on the condition that communication terminal 10B1 is operated by an administrator.

[0150] In step S906, terminal management unit 53 of communication management system 50, which has received the start response information from communication terminal 10B1, updates terminal management DB 5003 based on communication ID "01b1" of communication terminal 10B1 included in the start response information. For example, terminal management unit 53 changes the operating status information corresponding to communication ID "01b1" of communication terminal 10B1 to "ONLINE (communicating)" and updates the information on the reception date and time.

[0151] In step S907, session management unit 55 of communication management system 50 assigns a session ID, which is identification information for identifying the session. Session management unit 55 also stores the created session ID in session management DB 5005 in association with the communication ID of the request source terminal (communication ID of communication terminal 10A) and the communication ID of the destination terminal (communication ID of communication terminal 10B1).

[0152] In step S909, the session management unit 55 of the communication management system 50 transmits session information to the relay device 30. This session information includes, for example, information such as the session ID created in step S907. The relay device 30 can acquire the session information from the session management DB 5005 based on the session ID.

[0153] In step S910a, session management unit 55 of communication management system 50 transmits start instruction information to communication terminal 10A to instruct it to start a session. Similarly, in step S910b, session management unit 55 of communication management system 50 transmits start instruction information to communication terminal 10B1 to instruct it to start a session. The start instruction information includes a session ID, and communication terminal 10 can obtain session information from session management DB 5005 based on the session ID.

[0154] In step S911a, based on the received start instruction information, communication terminal 10A establishes a session with relay device 30. Similarly, in step S911b, based on the received start instruction information, communication terminal 10B1 establishes a session with relay device 30. This allows communication terminals 10A and 10B1 to participate in the same session.

[0155] In step S911c, wide-angle image transmission unit 31 of wide-angle imaging device 9 transmits equirectangular projection images to communication terminal 10B1. For example, when communication terminal 10B1 establishes a session, communication terminal 10B1 instructs wide-angle imaging device 9 to start capturing and transmitting equirectangular projection images. Alternatively, wide-angle imaging device 9 may constantly transmit equirectangular projection images to communication terminal 10B1 while wide-angle imaging device 9 is powered on. Furthermore, equirectangular projection images may be transmitted before a session is established.

[0156] Communication terminal 10A and communication terminal 10B1 can participate in a session with the same session ID and transmit and receive content data such as image data and audio data, thereby conducting, for example, a video conference (S912).

[0157] Furthermore, communication terminal 10A can use the established session to send and receive information about the operation of a device (for example, mobile device 20) to and from communication terminal 10B1.

[0158] In addition, the transmission and reception of information regarding the operation of the device may be performed using a control session via the communication management system 50 without using a session, or may be performed between the communication terminal 10A and the communication terminal 10B1 via the communication network 2, etc.

[0159] Here, the following description will be given assuming that communication terminal 10A uses the established session to transmit and receive information relating to the operation of a device (for example, mobile device 20) to and from communication terminal 10B1.

[0160] In step S913, when the operator of the communication terminal 10A inputs information into the operation screen of the device, operation instruction information corresponding to the input operation is transmitted using the session. This operation instruction information includes, for example, the communication ID of the communication terminal 10A and an operation instruction corresponding to the operation content of the operator.

[0161] In step S914, device control unit 19c of communication terminal 10B1 controls mobile device 20 via inter-device communication unit 19d based on the operation instruction included in the notified operation instruction information.

[0162] <Pattern 1> Next, for each of patterns 1 to 3, a display example will be described that allows the operator to know in which direction the mobile device 20 is facing when the operator is looking in a certain direction.

[0163] First, pattern 1 will be described using Figures 18 to 21. Figure 18 is an example of a diagram illustrating a reference imaging direction that serves as a reference for the imaging direction and the front direction of mobile device 20. Because wide-angle imaging device 9 is fixed to mobile device 20, the front direction of mobile device 20 can be expressed as the imaging direction of wide-angle imaging device 9. The imaging direction that is the same as the front direction of mobile device 20 is set as the reference imaging direction (ea0, aa0).

[0164] When the imaging direction in which the operator is looking is the standard imaging direction, the operator is looking at the front of the telepresence robot from behind the telepresence robot.

[0165] 19 is a diagram showing the relationship between an arbitrary imaging direction (ea, aa) displayed by the operator and the reference imaging direction (ea0, aa0). Since the reference imaging direction (ea0, aa0) is fixed, the difference (Δea, Δaa) between the imaging direction (ea, aa) and the reference imaging direction (ea0, aa0) is as follows: Δea=ea0-ea Δaa=aa0-aa To simplify the explanation, only the horizontal direction will be considered. When the imaging direction changes by Δaa from the reference imaging direction, it means that the telepresence robot has rotated in the opposite direction by Δaa from the perspective of the telepresence robot. For example, if the imaging direction increases by 90 degrees horizontally, the telepresence robot will rotate in a direction that decreases by 90 degrees horizontally. If the imaging direction decreases by 90 degrees horizontally, the telepresence robot will rotate in a direction that increases by 90 degrees horizontally. The center of rotation can be the center of gravity of the telepresence robot.

[0166] Therefore, the display control unit 16 rotates the telepresence robot by (-Δea, -Δaa) and displays it. (-Δea, -Δaa) may be calculated as the difference from the immediately preceding imaging direction (ea, aa) instead of the difference from the reference imaging direction (ea0, aa0).

[0167] The display control unit 16 rotates a pre-prepared polygonal model of the telepresence robot in the horizontal and elevation directions, and performs coordinate transformation to rotate the model around two axes of a Cartesian coordinate system that pass through the center of the polygonal model of the telepresence robot.

[0168] FIG. 20 is an example of a diagram illustrating perspective projection transformation. Perspective projection transformation is a transformation in which a viewpoint is placed on the Z axis and an object is projected onto a projection plane 830 perpendicular to the Z axis. The objects in this embodiment are a polygonal model of a telepresence robot and a predetermined area image Q of a celestial sphere image. The origin O is the center of the three-dimensional sphere as shown in FIG. 10. The objects closest to the origin O are the projection plane 830, the polygonal model 839 of the telepresence robot, and the predetermined area image Q. In normal perspective projection transformation for displaying a celestial sphere image on the display unit 109, the polygonal model 839 of the telepresence robot is not positioned, but in pattern 1, the polygonal model 839 of the telepresence robot is positioned on the Z axis. The distance from the projection plane 830 to the projection plane 830 may be determined experimentally. A wide-angle imaging device 9 is mounted on the upper side of communication terminal 10A, and the omnidirectional image is captured by wide-angle imaging device 9, so that the deviation between the viewpoint and wide-angle imaging device 9 is reduced when polygon model 839 is closer to projection surface 830.

[0169] It should be noted that the polygon model 839 of the telepresence robot does not have to be on the Z axis. For example, it may be arranged so as to be projected onto one of the four corners of the predetermined area image Q.

[0170] Polygon model 839 of the telepresence robot is rotated in the local coordinate system by the difference (Δea, Δaa) between the imaging direction (ea, aa) and the reference imaging direction (ea0, aa0) as described in Fig. 19. Then, it is placed in a coordinate system for projecting a celestial sphere video as shown in Fig. 20. This makes it possible to display in an image of the telepresence robot the direction in which mobile device 20 is facing in the celestial sphere video.

[0171] FIG. 21 shows an example of a celestial sphere image display screen 1010 displayed in pattern 1. On the celestial sphere image display screen 1010 in FIG. 21(a), communication terminal 10A displays a celestial sphere image in the reference imaging direction. Therefore, image 840 of the telepresence robot faces away from the operator. Next, the operator rotates the celestial sphere image to the right. For convenience of explanation, it is assumed that the celestial sphere image is rotated by 90 degrees horizontally. During this time, mobile device 20 does not move or turn.

[0172] 21(b) shows an image 840 of the telepresence robot after the operator rotates the omnidirectional image to the right. Since the telepresence robot in the real space is facing 90 degrees to the left with respect to the imaging direction of the omnidirectional image in FIG. 21(b), the image 840 of the telepresence robot also faces 90 degrees to the left. When the predetermined area image Q changes, the orientation of the image 840 of the telepresence robot changes according to the amount of change.

[0173] Therefore, even if the operator rotates the omnidirectional image as desired, they can easily grasp the direction in which the telepresence robot in the real space is facing.

[0174] Fig. 22 is an example of a flowchart showing a procedure in which communication terminal 10A displays a predetermined area image on display unit 109. The process in Fig. 22 is executed every time the user changes the imaging direction.

[0175] First, the operation input receiving unit 12 receives an operation of the imaging direction by the operator (S1001).

[0176] Next, the display control unit 16 calculates the difference (Δea, Δaa) between the imaging direction and the reference imaging direction (S1002). Instead of using the reference imaging direction as a reference, the difference from the immediately preceding imaging direction may be calculated.

[0177] Next, the display control unit 16 rotates the polygon model of the telepresence robot by (-Δea, -Δaa) in the opposite direction to the difference (S1003).

[0178] Then, the display control unit 16 performs perspective projection conversion on the predetermined area image and the polygon model and displays them (S1004).

[0179] It is preferable that the image of the telepresence robot is semi-transparent. It is also preferable that the degree of semi-transparency can be determined by the operator. Furthermore, it is also preferable that the operator can set whether to display the image of the telepresence robot. This prevents the image of the telepresence robot from interfering with the viewing of the omnidirectional video.

[0180] <Pattern 2> Next, we will explain the method of displaying the direction of the telepresence robot in Pattern 2. Pattern 2 is a display method in which, separate from the spherical image viewed by the operator, the direction in which the spherical image is displayed as seen from the telepresence robot is displayed by the telepresence robot and an imaging direction arrow 8.

[0181] FIG. 23 shows an example of a polygon model of a telepresence robot and an imaging direction arrow object. A polygon model 850 of the telepresence robot is prepared in advance. The polygon model 850 of the telepresence robot is placed in a predetermined coordinate system. The imaging direction arrow object 851 is also formed from a polygon. Since the imaging direction determined by the operator is represented by ea and aa, the display control unit 16 generates an imaging direction arrow object 851 of a predetermined length indicating the imaging direction ea and aa. For example, the prepared imaging direction arrow object 851 is placed facing the imaging direction ea and aa.

[0182] As shown in the figure, the display control unit 16 aligns the arrowhead of the imaging direction arrow object 851 with the wide-angle imaging device 9 and places it in a predetermined coordinate system. After that, it is sufficient to perform perspective projection transformation and place it on the display unit 109. This makes it possible to display the imaging direction arrow 8 three-dimensionally. The imaging direction arrow 8 is information indicating in which direction the predetermined area image Q is an image as seen from the mobile device 20. Therefore, the imaging direction arrow 8 is an image of an arrow whose origin is the telepresence robot image 854 (mobile device 20).

[0183] 2(b), the telepresence robot image 854 and the imaging direction arrow 8 are displayed alongside the predetermined area image 853. However, to accommodate cases where the screen size of the display unit 109 is small, the display control unit 16 may display only one of them. For example, when the operator drags the predetermined area image 853 with a mouse or swipes with a finger, the telepresence robot image 854 and the imaging direction arrow 8 appear.

[0184] 23, perspective projection may be performed after rotating the telepresence robot object 851 according to the imaging direction. This is because the viewpoint from which the imaging direction arrow 8 is easily seen varies depending on the imaging direction. For example, the telepresence robot object 851 in the standard imaging direction (at a horizontal angle that makes the imaging direction arrow 8 facing forward easy to see) is rotated horizontally by Δea. This makes it easier for the operator to understand in which direction the imaging direction arrow 8 is pointing, regardless of the imaging direction.

[0185] As shown in Fig. 24 , in pattern 2, the imaging direction arrow 8 may be displayed separately for the horizontal direction and the elevation angle direction. Fig. 24 shows another display example of the imaging direction arrow 8 in pattern 2. The omnidirectional image display screen 1010 in Fig. 24(a) has a omnidirectional image field 1001, a horizontal direction instruction field 1003, and an elevation angle direction instruction field 1004. The horizontal direction instruction field 1003 has an image 855 of a telepresence robot viewed from above, and indicates only the horizontal direction of the imaging direction with a horizontal direction arrow 8a. The elevation angle direction instruction field 1004 has an image 854 of the telepresence robot, and indicates only the elevation angle direction of the imaging direction with an elevation angle direction arrow 8b.

[0186] By separately displaying the horizontal direction and the elevation angle direction, the operator can easily determine the horizontal extent and the elevation angle extent of the imaging direction.

[0187] In Figure 24(a), the horizontal direction of the imaging direction is included in the forward 180 degrees, so the elevation angle direction arrow 8b in the elevation angle direction indication field 1004 is displayed in front of the telepresence robot. On the other hand, in Figure 24(b), the horizontal direction of the imaging direction is included in the backward 180 degrees, so the elevation angle direction arrow 8b in the elevation angle direction indication field 1004 is displayed behind the telepresence robot. By switching the display position of the elevation angle direction arrow in this way, the operator can intuitively know whether the horizontal direction is forward or backward.

[0188] In addition, since the images 854, 855 of the telepresence robot, the horizontal arrow 8a, and the elevation arrow 8b in Figure 24 represent directions in two dimensions, there is no need to prepare a polygon model and perform perspective projection transformation, which reduces the processing load.

[0189] Fig. 25 is an example of a flowchart illustrating a procedure in which communication terminal 10A displays a spherical video on display unit 109 in pattern 2. The process in Fig. 25 is executed every time the user changes the imaging direction.

[0190] First, the operation input receiving unit 12 receives an operation of the imaging direction by the operator (S2001).

[0191] Next, the display control unit 16 performs perspective projection conversion on the predetermined area image and displays it (S2002).

[0192] Next, the display control unit 16 determines the viewpoint of the polygon model 850 of the telepresence robot according to the imaging direction (S2003).

[0193] Next, the display control unit 16 places the imaging direction arrow object 851 in a predetermined coordinate system (S2004). That is, the display control unit 16 aligns the arrowhead of the imaging direction arrow object 851 with the wide-angle imaging device 9, and places it in the coordinate system in which the telepresence robot object is placed.

[0194] The display control unit 16 performs perspective projection transformation on the polygon model 850 of the telepresence robot and the imaging direction arrow object 851, and displays them on the display unit 109 (S2005).

[0195] <Pattern 3> In pattern 3, the operation button is displayed only when the imaging direction substantially coincides with the front direction of mobile device 20. That is, display control unit 16 displays operation button 860 only when imaging direction (ea, aa)≈(ea0, aa0). See FIG. 2(c) for the spherical video display screen 1010 in pattern 3.

[0196] Fig. 26 is an example of a flowchart illustrating a procedure in which communication terminal 10A displays a spherical video on display unit 109 in pattern 3. The process in Fig. 26 is executed every time the user changes the imaging direction.

[0197] First, the operation input receiving unit 12 receives an operation of the imaging direction by the operator (S3001).

[0198] Next, the display control unit 16 performs perspective projection transformation on the predetermined area image Q and displays it (S3002).

[0199] Next, the display control unit 16 determines whether the imaging direction and the reference imaging direction substantially coincide with each other (S3003).

[0200] If the determination in step S3003 is Yes, the display control unit 16 displays the operation button 860 superimposed on the spherical image (S3004).

[0201] If the determination in step S3003 is No, the display control unit 16 does not display the operation button 860 on the spherical image.

[0202] Therefore, the operation button 860 is displayed only when the front direction of the mobile device 20 and the imaging direction of the omnidirectional video image viewed by the operator are made to substantially coincide with each other, allowing the operator to know where the mobile device 20 will move when moving forward. In addition, when the operation button 860 is displayed, the operator can know the orientation of the mobile device 20.

[0203] Note that, although the operation button 860 is not displayed at all in the left diagram c-1 of FIG. 2(c), when the front direction of the mobile device 20 and the imaging direction of the omnidirectional video viewed by the operator do not substantially coincide, the operation button 860 may be displayed in a manner that the operator cannot operate. The manner in which the operation button 860 cannot be operated means, for example, displaying it in low brightness, displaying it in gray, or displaying it semi-transparently (increasing transparency). Even if the operator presses the operation button 860, the operation input receiving unit 12 does not accept the operation.

[0204] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0205] For example, in this embodiment, a front image (or a rear image) captured by a front imaging device is not described, but a front image (or a rear image) is displayed as appropriate in each of patterns 1 to 3.

[0206] 27 shows an example of a simultaneous video display screen 2001 that simultaneously displays a spherical video and a front video (or a rear video). The simultaneous video display screen 2001 has an overall display field 2002 and a front display field 2003. An image of any one of patterns 1 to 3 is displayed in the overall display field 2002. The front video (or rear video) captured by the front imaging device is displayed in the front display field 2003. Therefore, the operator can display any direction in the overall display field 2002 while always checking the front or rear.

[0207] In this embodiment, wide-angle imaging device 9 transmits equirectangular projection images to communication terminal 10B, and communication terminal 10B transmits the equirectangular projection images to communication terminal 10A, but wide-angle imaging device 9 may directly connect a session with communication terminal 10A and transmit the equirectangular projection images. In this case, wide-angle imaging device 9 becomes one form of communication terminal 10, which transmits images but does not receive images.

[0208] Although the present embodiment has been described as transmitting a rear view image, communication terminal 10B does not necessarily have to transmit a rear view image. Communication terminal 10B may also transmit an image with a normal angle of view separately from the front view image and the rear view image.

[0209] Furthermore, in this embodiment, the mobile device 20 that travels on land has been described as an example, but the mobile device 20 may also fly in the air, such as a drone. It may also sail on the sea or travel underwater. It may also travel underground, through narrow passages, or underground. When traveling on land, it may move on wheels, or it may move on multiple legs such as two, three, or four legs, or it may move on caterpillars (registered trademark).

[0210] Furthermore, the display method of this embodiment may be applied not only to telepresence robots but also to images from the wide-angle imaging device 9 and planar imaging device mounted on a device that is moved by a person.

[0211] Furthermore, in this embodiment, an example in which one operator operates one mobile device 20 has been mainly described, but it is also possible for one operator to operate multiple mobile devices 20. In this case, communication terminal 10A establishes sessions with multiple communication terminals 10B. Conversely, it is also possible for multiple operators to operate one mobile device 20. In this case, for example, the operation instruction information last transmitted to communication terminal 10B may be valid, or the concept of operation authority may be introduced so that only communication terminal 10A with operation authority is allowed to transmit operation instructions. Operation authority can be transferred by an operation by the operator or the like.

[0212] Furthermore, in the present embodiment, the operation instruction information is transmitted in a communication session, but it may be transmitted in a content data session.

[0213] In addition, in this embodiment, the communication terminal 10 communicates via the relay device 30, but the communication terminal 10 may communicate without the relay device. For example, WebRTC (Web Real-Time Communication) is known as a communication protocol for such communication.

[0214] 14 and other configuration examples are divided according to main functions to facilitate understanding of the processing by the communication management system 50 and the communication terminal 10. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the communication management system 50 and the communication terminal 10 can be divided into even more processing units depending on the processing content. Furthermore, the processing can be divided so that one processing unit includes even more processes.

[0215] Also, in this embodiment, for convenience of explanation, the communication management system 50 and the relay device 30 are described as separate devices, but a device that integrates the functions of both the communication management system 50 and the relay device 30 may provide the functions of the communication management system 50 and the relay device 30.

[0216] Furthermore, the communication system 1 may have a plurality of communication management systems 50, and the functions of the communication management systems 50 may be distributed and installed on a plurality of servers.

[0217] Furthermore, one or more of the databases that the communication management system 50 has in the storage unit 5000 may exist on the communication network 2.

[0218] The transmitting / receiving unit 11 is an example of a communication means, the display control unit 16 is an example of a display processing means, and the operation input receiving unit 12 is an example of a receiving means. [Explanation of symbols]

[0219] 1. Communication Systems 9 Wide-angle imaging device 10. Communication terminals 20 Mobile Devices 50 Communication Management System [Prior art documents] [Patent documents]

[0220] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-112030

Claims

1. A communication terminal that communicates with another communication terminal mounted on a mobile device via a network, a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal; a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal; and the display processing means displays an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image allowing the orientation of the mobile device to be grasped; When at least a part of the wide-angle image displayed on the display unit changes, the image of the mobile device displayed in a manner that allows the orientation of the mobile device to be known changes orientation in accordance with the change. A communication terminal characterized by:

2. A communication terminal that communicates with another communication terminal mounted on a mobile device via a network, a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal; a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal; and The display processing means displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped; In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the mobile device is displayed together with the image of the mobile device. A communication terminal characterized by:

3. the communication means transmits operation instruction information for the mobile device to the other communication terminal; 3. The communication terminal according to claim 1 or 2.

4. The communication terminal described in claim 1 or 2, characterized in that the display processing means superimposes an image of the mobile device displayed in a manner that makes the orientation of the mobile device clear on at least a portion of the wide-angle image displayed on the display unit.

5. The communication terminal according to claim 1, characterized in that the display processing means performs perspective projection conversion of the wide-angle image attached to a three-dimensional sphere and a model of the mobile device represented in a manner that allows the orientation of the mobile device to be seen, thereby superimposing an image of the mobile device represented in a manner that allows the orientation of the mobile device to be seen on at least a portion of the wide-angle image displayed on the display unit.

6. The communication terminal described in claim 2, characterized in that the information indicating the direction in which at least a portion of the wide-angle image displayed on the display unit is an image of an arrow starting from the image of the mobile device.

7. The communication terminal of claim 6, characterized in that the display processing means displays information indicating the direction in which at least a portion of the wide-angle image displayed on the display unit is an image starting from the image of the mobile device by performing perspective projection transformation on a model of the mobile device and a model of an arrow indicating the direction in which at least a portion of the wide-angle image displayed on the display unit is an image starting from the image of the mobile device.

8. The communication terminal described in claim 2, characterized in that the information indicating the direction in which at least a portion of the wide-angle image displayed on the display unit is an image starting from the image of the mobile device is an image of arrows in the horizontal and elevation directions starting from the image of the mobile device.

9. The communication terminal described in claim 1 or 2, characterized in that the information regarding the orientation of the mobile device is an operation button for accepting operation instruction information for the mobile device, which is displayed when the center direction of at least a portion of the wide-angle image displayed on the display unit approximately coincides with the front direction of the mobile device.

10. A communication terminal that communicates with a telepresence robot via a network, the communication terminal comprising: a mobile device; another communication terminal mounted on the mobile device; and a wide-angle imaging device mounted on the mobile device or the other communication terminal and configured to capture wide-angle images, a communication means for receiving the wide-angle image captured by the wide-angle imaging device; a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal; and the display processing means displays an image of the telepresence robot on the display unit on which at least a part of the wide-angle image is displayed, the image enabling the orientation of the telepresence robot to be grasped; When at least a part of the wide-angle image displayed on the display unit changes, the image of the telepresence robot displayed in a manner that allows the orientation of the telepresence robot to be known changes orientation in accordance with the change. A communication terminal characterized by:

11. A communication terminal that communicates with a telepresence robot via a network, the communication terminal comprising: a mobile device; another communication terminal mounted on the mobile device; and a wide-angle imaging device mounted on the mobile device or the other communication terminal and configured to capture wide-angle images, a communication means for receiving the wide-angle image captured by the wide-angle imaging device; a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal; and The display processing means displaying an image of the telepresence robot on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the telepresence robot to be grasped; In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the telepresence robot is displayed together with the image of the telepresence robot. A communication terminal characterized by:

12. A communication terminal that communicates with other communication terminals mounted on a mobile device via a network, a communication means for receiving a wide-angle image captured by a wide-angle imaging device connected to the mobile device or the other communication terminal; a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal; It functions as The display processing means displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped; When at least a part of the wide-angle image displayed on the display unit changes, the image of the mobile device displayed in a manner that allows the orientation of the mobile device to be known changes orientation in accordance with the change. program.

13. A communication terminal that communicates with other communication terminals mounted on a mobile device via a network, a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal; a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal; It functions as The display processing means displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped; In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the mobile device is displayed together with the image of the mobile device. program.

14. A display method performed by a communication terminal that is mounted on a mobile device and communicates with another communication terminal via a network, comprising: a communication step in which a communication means receives a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal; a display processing step in which a display processing means displays at least a part of the wide-angle image received by the communication means in the communication step on a display unit of the communication terminal different from the other communication terminal; Including, The display processing step includes: displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped; When at least a part of the wide-angle image displayed on the display unit changes, the image of the mobile device displayed in a manner that allows the orientation of the mobile device to be known changes orientation in accordance with the change. A display method characterized by:

15. A display method performed by a communication terminal that is mounted on a mobile device and communicates with another communication terminal via a network, comprising: a communication step in which a communication means receives a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal; a display processing step in which a display processing means displays at least a part of the wide-angle image received by the communication means in the communication step on a display unit of the communication terminal different from the other communication terminal; Including, The display processing step includes: displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped; In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the mobile device is displayed together with the image of the mobile device. A display method characterized by:

16. A communication terminal that communicates with other communication terminals mounted on a mobile device via a network, a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal; a display processing unit that displays at least a part of the wide-angle image received by the communication unit on a display unit of the communication terminal different from the other communication terminal; It functions as The display processing means displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped; When at least a part of the wide-angle image displayed on the display unit changes, the image of the mobile device displayed in a manner that allows the orientation of the mobile device to be known changes orientation in accordance with the change. A computer-readable recording medium on which a program is recorded.

17. A communication terminal that communicates with other communication terminals mounted on a mobile device via a network, a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal; a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal; It functions as The display processing means displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped; In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the mobile device is displayed together with the image of the mobile device. A computer-readable recording medium on which a program is recorded.

Images