Communication system, communication method, program

Abstract

To provide a communication system which can determine whether to permit an operation according to the role of a communication terminal.SOLUTION: In the communication system, communication through a communication management system 50 is performed among a first communication terminal 10B which is loaded on a moving device 20, a second communication terminal 10A which operates the moving device, and a plurality of third communication terminals 10C. Each of the first communication terminal, the second communication terminal and the plurality of third communication terminals includes a communication unit 11 which transmits the role of the self-device to the communication management system. The communication management system determines whether the operation right of the moving device is given only to the predetermined communication terminal of the role, or alternatively, whether the first communication terminal, having received an operation command and the role, accepts an operation according to the role.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a communication system, a communication method, and a program.

Background Art

[0002] Communication systems in which communication terminals communicate between multiple sites via a communication network such as the Internet or a LAN (Local Area Network) to transmit and receive video and audio to and from each other are widespread. Using a communication system, users in remote locations can communicate, for example, hold a video conference.

[0003] In addition, a telepresence robot in which a communication system is applied to a robot is known. A telepresence robot is a robot that combines "video conferencing" and "remote operation technology". It is a technology that combines remote control from a remote location and robot technology to operate a moving device that moves a robot from a distance and behaves as if an operator is present (has a presence) at a certain location.

[0004] In addition, telepresence robots are also expected to be used for the purpose of autonomous support for disabled people. In such a use case, in order to assist the robot operated by a disabled person, assistants often wait beside the robot and beside the disabled person. Regarding the technology of operating a telepresence robot, there is also disclosed a technology in which a user who does not have the operation authority for a specific function of a remote control robot remotely operates the specific function as needed (see, for example, Patent Document 1). Patent Document 1 discloses a method of permitting remote operation of a specific function only to a user having the operation authority for the specific function among a plurality of users remotely operating a telepresence robot.

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, conventional communication systems have had the problem that they cannot determine whether to permit an operation according to the role of the communication terminal. First, in conventional telepresence robots, remote communication can be carried out only between the telepresence robot and the operator. For this reason, conventionally, assistants have had to connect terminal devices to another remote communication system or communicate with each other using a mobile phone or the like to communicate. However, if another communication terminal including an assistant communicates with the telepresence robot, there is a risk that a communication terminal other than the operator will operate the telepresence robot.

[0006] In view of the above problems, an object of the present invention is to provide a communication system that can determine whether to permit an operation according to the role of the communication terminal.

Means for Solving the Problems

[0007] In view of the above problems, the present invention is a communication system in which a first communication terminal mounted on a mobile device, a second communication terminal that operates the mobile device, and a plurality of third communication terminals communicate via a communication management system, wherein the first communication terminal, the second communication terminal, and the plurality of third communication terminals each have a communication unit that transmits its own role to the communication management system, and the communication management system determines whether to give the operation right of the mobile device only to a communication terminal of a predetermined role, or whether the first communication terminal that has received an operation command and the role accepts the operation according to the role. The first communication terminal has an imaging unit, and the first communication terminal transmits an image captured by the imaging unit to the second communication terminal and the third communication terminal, and the first communication terminal transmits at least one of a front image and a back image captured by the imaging unit to the second communication terminal and the third communication terminal. It is characterized by this.

Effects of the Invention

[0008] A communication terminal that can determine whether to permit an operation according to the role of the communication terminal can be provided.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0010] Hereinafter, as an example of an embodiment for carrying out the present invention, a communication system and a communication method performed by the communication system 1 will be described with reference to the drawings.

Examples

[0011] <Outline of Operations of Communication System and Communication Terminal> First, the outline of the communication system 1 will be described with reference to FIG. 1. FIG. 1(a) shows an example of a schematic configuration diagram of the communication system 1 using a telepresence robot. A telepresence robot is arranged in the company 801, and there is an in-company assistant 842. At home 802, there are an operator 810 and a home assistant 841. The telepresence robot has a communication terminal 10B (an example of a first communication terminal) mounted on a moving device 20. Also, the operator 810 has a communication terminal 10A (an example of a second communication terminal).

[0012] The communication terminals 10A and 10B each have a planar imaging device and a microphone, and transmit and receive video and audio to and from each other. That is, the communication terminal 10B transmits a front video 831 (or / and a rear video 832) to the communication terminal 10A, and the communication terminal 10A transmits the front video 831 to the communication terminal 10B.

[0013] Also, in the communication terminal 10A, application software (hereinafter referred to as an app) that accepts operations related to the movement of the moving device 20 and the like is operating, and the operator 810 inputs an operation command while checking the video transmitted by the communication terminal 10B. The operation command input by the operator 810 is transmitted from the communication terminal 10A to the communication terminal 10B. Also, the home assistant 841 has a communication terminal 10C1 (non-operating terminal), and the in-company assistant 842 has a communication terminal 10C2 (non-operating terminal). The non-operating terminals can also communicate with each other via the communication network 2.

[0014] The communication terminal 10B and the moving device 20 can communicate via short-range wireless such as Bluetooth (registered trademark), and the communication terminal 10B controls the moving device 20 based on the operation command, so the operator 810 can move the moving device 20 from a remote location (home).

[0015] By moving the mobile device 20 in this way, as shown in Fig. 1(b) (the home caregiver 841 and the in-company caregiver 842 are omitted), the operator 810 can talk to the desired partner 820. In Fig. 1(b), the operator 810 moves the mobile device 20 to the location of Mr. A 820a and talks to Mr. A, and then moves the mobile device 20 to the location of Mr. B 820b and talks to Mr. B.

[0016] Not only meeting people, but also the operator 810 can do many things remotely without moving to the site, such as checking the products of a store in a remote location, inspecting a factory, or participating in an exhibition.

[0017] <Regarding terms> The operator 810 is the person who operates the mobile device 20. Although the operator 810 is also the user who uses the communication terminal 10A, in this embodiment, the operator is referred to as such.

[0018] The conversation partner 820 is a person within the company or the like, but may also be an animal such as a dog or a cat. Also, the mobile device 20 does not have to be used for the operator 810 to talk to the partner. For example, when the mobile device 20 patrols within the company or the like, the communication terminal 10B may transmit video and audio to the communication terminal 10A, and there may be cases where the operator does not talk.

[0019] The place where the mobile device 20 moves may be any place where the mobile device 20 can move. Also, it is not limited to a device that moves by power such as a motor, and a person may assist the movement supplementarily.

[0020] A caregiver refers to a person who provides assistance and support in terms of health and life so that a person with physical or mental disabilities who has difficulty in leading a daily life can lead a physically, mentally, and socially independent life.

[0021] <Example of system configuration> FIG. 2 is a diagram showing a configuration example of a communication system 1 according to an embodiment. The communication system 1 includes a plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E, 10C1, 10C2), a mobile device 20, a relay device 30, a communication management system 50, and the like. In the following description, when indicating an arbitrary communication terminal among the plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E, 10C1, 10C2), "communication terminal 10" is used. Also, when indicating an arbitrary communication terminal among the communication terminals (10B1, 10B2, 10B3), "communication terminal 10B" is used. Further, when indicating an arbitrary communication terminal among the communication terminals (10C1, 10C2), "communication terminal 10C (an example of a third communication terminal)" is used. Also, the number of communication terminals 10 in FIG. 2 is an example.

[0022] The communication terminal 10, the relay device 30, and the communication management system 50 are each communicably connected via a communication network 2 to other communication terminals 10, devices, and systems. The communication network 2 may include, for example, a LAN (Local Area Network), the Internet, a mobile phone network, or a dedicated line.

[0023] The communication terminal 10 is, for example, a general-purpose information processing device such as a tablet terminal, a smartphone, a PC (Personal Computer), or a dedicated videoconference device. The communication terminal 10 can perform, for example, a videoconference by transmitting and receiving, for example, image data, audio data, etc. to and from one or more other communication terminals 10.

[0024] By executing an application corresponding to the communication system 1, the communication terminal 10A can hold a video conference with the communication terminal 10B and remotely operate the mobile device 20 via the communication terminal 10B. For example, the communication terminal 10A can move the mobile device 20 mounted on the communication terminal 10B, such as forward, backward, left, or right, by operating the operation buttons displayed on the display screen of the video conference. Note that the communication terminal 10A can also operate browser software, and the browser software can display video and receive operations of the mobile device 20 on the browser software.

[0025] The mobile device 20 and the communication terminal 10B are called a telepresence robot. The mobile device 20 is a device having a traveling function that moves, such as "forward", "backward", "right turn", "left turn", etc., by driving a plurality of wheels, for example, in response to control from the communication terminal 10B attached to the mobile device 20. When the user instructs a left or right movement, it moves forward after a "right turn" or a "left turn". Note that the appearance of the mobile device 20 shown in FIG. 2 is merely an example. The mobile device 20 only needs to be movable together with the communication terminal 10B in response to an operation command from the communication terminal 10B mounted on the mobile device 20.

[0026] By executing an application corresponding to the communication system 1, the communication terminals 10C1 and 10C2 hold a video conference with other communication terminals 10.

[0027] The relay device 30 is, for example, an information processing device or a system including one or more information processing devices, and relays content data (video, audio, operation instruction information) transmitted and received between a plurality of communication terminals 10. However, a plurality of communication terminals 10 may directly transmit and receive content data without passing through the relay device 30.

[0028] The communication management system 50 is, for example, an information processing device or a system including one or more information processing devices. The communication management system 50 performs, for example, login authentication from the communication terminal 10, management of the communication status of the communication terminal 10, management of a destination list, etc., and control of a session for communication between a plurality of communication terminals 10 via the relay device 30.

[0029] In one embodiment, the session is realized by the relay device 30 relaying content data including image data and sound data (voice and other sounds) between a plurality of communication terminals 10.

[0030] In the above configuration, for example, an operator of the communication terminal 10A can communicate with the communication terminal 10B and remotely operate the communication terminal 10B and the mobile device 20 to move them. Thereby, the operator of the communication terminal 10A can move the communication terminal 10B and the mobile device 20 near an arbitrary partner to hold a video conference or the like.

[0031] Further, the communication system 1 includes a data providing system that transmits content data unidirectionally from one communication terminal 10 to another communication terminal 10 via the communication management system 50, and a communication system that mutually transmits information, emotions, etc. between a plurality of communication terminals 10 via the communication management system 50. This communication system is a system for mutually transmitting information, emotions, etc. between a plurality of communication terminals via the relay device 30, and examples include a television (or video) conference system and a videophone system.

[0032] <Hardware Configuration> <<Hardware Configuration of Communication Terminal>> FIG. 3 is a diagram showing an example of the hardware configuration of a communication terminal according to an embodiment of the present invention. The communication terminal 10 includes a configuration of a general computer, and for example, has a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a flash memory 104, and an SSD (Solid State Drive) 105, etc. Further, the communication terminal 10 has a media I / F (Interface) 107, an input unit 108, a display unit 109, a network I / F 111, a camera 112, an image sensor I / F 113, a microphone 114, a speaker 115, and an audio input / output I / F 116, etc. Furthermore, the communication terminal 10 has an external device connection I / F 117, a short-range wireless communication unit 118, and a bus 119, etc.

[0033] The CPU 101 is, for example, an arithmetic device that reads programs and data from the ROM 102, the flash memory 104, etc. and executes processing to realize each function provided in the communication terminal 10. The ROM 102 is a non-volatile memory that stores in advance programs, etc. used for starting up the CPU 101, such as an IPL (Initial Program Loader). The RAM 103 is a volatile memory used as a work area, etc. of the CPU 101.

[0034] The flash memory 104 is, for example, a storage device that stores an OS (Operating System), application programs, and various data, etc. The SSD 105 controls the reading and writing of various data with respect to the flash memory 104 according to the control of the CPU 101. The media I / F 107 controls the reading or writing (storage) of data with respect to a recording medium 106, which is a recording medium such as a memory card. A program for controlling the communication terminal 10 may be stored in the recording medium.

[0035] The input unit 108 is an input device for receiving input operations from an operator, such as a touch panel, a keyboard, a pointing device, etc. Voice input may also be possible. The display unit 109 is a display device that performs various displays for the operator. Note that the input unit 108 and the display unit 109 may be a display input unit 110, such as a touch panel display in which a touch panel and a display are integrated.

[0036] The network I / F 111 is a communication interface for the communication terminal 10 to transmit and receive data using the communication network 2. The camera 112 includes an imaging element for imaging a subject according to the control of the CPU 101. The imaging element I / F 113 controls the imaging by the camera 112 and converts the imaged data into predetermined image data. Although the camera 112 is a planar imaging device, it may be a wide-angle imaging device that can image 360 degrees around.

[0037] The microphone 114 converts the recorded voice into an electrical signal. The speaker 115 converts the voice signal into voice and outputs it. The voice input / output I / F 116 controls the input / output of voice by the microphone 114 and the speaker 115.

[0038] The external device connection I / F 117 is an interface for connecting external devices, such as USB (Universal Serial Bus). The external devices include, for example, the mobile device 20 in FIG. 2.

[0039] The short-range wireless communication unit 118 is a communication interface for communicating with an external device (e.g., the mobile device 20) by short-range wireless communication such as Bluetooth (registered trademark) or Bluetooth (registered trademark) Low Energy. The bus 119 is commonly connected to the above-described components and transmits address signals, data signals, various control signals, etc.

[0040] <<Hardware Configuration of Communication Management System, Relay Device>> FIG. 4 is a diagram showing a hardware configuration example of a communication management system 50 according to an embodiment. The communication management system 50 has a configuration of a general computer and includes, for example, a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD (Hard Disk Drive) 505, a media drive 507, a display 508, and the like. The communication management system 50 also includes a network I / F 509, a keyboard 511, a mouse 512, a CD-ROM drive 514, a bus 510, and the like.

[0041] The CPU 501 is an arithmetic unit that realizes each function of the communication management system 50 by reading programs and data stored in, for example, the ROM 502, the HD 504, etc. and executing processes. The ROM 502 is a non-volatile memory that stores programs used for starting up the CPU 501, such as an IPL, in advance. The RAM 503 is a volatile memory used as a work area of the CPU 501, etc.

[0042] The HD 504 is a storage device that stores, for example, programs such as an OS and application programs, and various data. The HDD 505 controls reading and writing of various data to and from the HD 504 according to the control of the CPU 501. The display 508 is a display device that displays various information such as a cursor, a menu, a window, characters, or an image.

[0043] The network I / F 509 is a communication interface for performing data communication using the communication network 2. The keyboard 511 is an example of an input device that receives input operations such as characters, numerical values, and various instructions by a system administrator. The mouse 512 is an example of a pointing device that receives operations such as selection and execution of various instructions, selection of a processing target, and movement of a cursor by a system administrator.

[0044] Regarding the display 508, keyboard 511, and mouse 512, the communication management system 50 or the relay device 30 may not have them normally. In this case, they may be connected as needed.

[0045] The media drive 507 controls, for example, the reading and writing (recording) of data with respect to the recording medium 506 such as a memory card. The CD-ROM drive 514 controls the reading or writing of data with respect to the disk 513 as an example of a removable recording medium. The bus 510 electrically connects the above-described respective components and transmits an address signal, a data signal, various control signals, and the like.

[0046] Note that the above-described hardware configuration of the computer is merely an example.

[0047] It is assumed that the relay device 30 has the same hardware configuration as the communication management system 50. Note that the programs for the communication terminal 10, the relay device 30, and the communication management system 50 may be recorded on a computer-readable recording medium in an installable format or an executable format file and distributed. Further, examples of the above-described recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc, a USB memory, and the like. Also, a recording medium such as a CD-ROM on which each program is stored, and the HD 504 on which these programs are stored can be provided as a program product domestically or abroad.

[0048] <<Hardware Configuration of Mobile Device>> FIG. 5 is a diagram showing a hardware configuration example of a mobile device according to an embodiment. The mobile device 20 includes, for example, a CPU 401, a RAM 402, a ROM 403, an external device I / F 404, a short-range wireless communication unit 405, a wheel drive unit 406, a steering unit 407, and the like.

[0049] The CPU 401 is an arithmetic unit that realizes each function of the mobile device 20 by executing programs stored in the ROM 403 and the like. The RAM 402 is a volatile memory used as a work area of the CPU 401 and the like. The ROM 403 is a non-volatile memory that stores programs and the like of the mobile device 20. The ROM 403 may be a rewritable non-volatile memory such as a flash ROM, for example.

[0050] The external device I / F 404 is a wired communication interface for performing communication by making a wired connection with the external device connection I / F 117 of the communication terminal 10 and the like.

[0051] The short-range wireless communication unit 405 is a wireless communication interface for performing wireless communication by the same wireless communication method as the short-range wireless communication unit 118 of the communication terminal 10, for example. The mobile device 20 may be capable of communicating with the communication terminal 10 by the external device I / F 404 or the short-range wireless communication unit 405, for example.

[0052] The wheel drive unit 406 is an example of a drive device that drives wheels for moving the mobile device 20. The wheel drive unit 406 includes, for example, a motor or the like.

[0053] The steering unit 407 is an example of a steering device that steers the mobile device 20 that moves by the wheel drive unit 406. The steering unit 407 may be, for example, something that changes the direction or inclination of the wheels, or something that changes the direction of the mobile device 20 by controlling the rotational speed, speed, etc. of the left and right wheels.

[0054] <Regarding functions> Next, the functions of the communication system 1 will be described with reference to FIG. 6. FIG. 6 is a functional block diagram showing the functions of the communication management system 50, the communication terminal 10B, the communication terminal 10A, and the communication terminal 10C included in the communication system 1 in a block form.

[0055] <<Functional configuration of communication terminal 10A>> The communication terminal 10A receives, for example, operation commands for controlling devices such as the mobile device 20, but it is a communication terminal 10 that does not have a control function for devices such as the mobile device 20. The communication terminal 10A includes a communication unit 11, an operation input reception unit 12, a communication control unit 13, an imaging unit 14, an audio input unit 15a, an audio output unit 15b, a display control unit 16, and a storage / reading unit 17. Each of these units is a function or means realized by any of the components shown in FIG. 3 operating according to instructions from the CPU 101 according to a program for the communication terminal 10 expanded from the flash memory 104 onto the RAM 103. Further, the communication terminal 10 has a storage unit 18 realized by the RAM 103 shown in FIG. 3 and the flash memory 104 shown in FIG. 3.

[0056] <<Functional Configuration of Communication Terminal 10B>> The communication terminal 10B is a communication terminal mounted on the mobile device 20. The communication terminal 10B is, for example, an example of a communication terminal having a control function for devices such as the mobile device 20. In addition to the functional configuration of the communication terminal 10A described above, the communication terminal 10B includes an operation instruction reception unit 19a, a device control unit 19c, an inter-device communication unit 19d, and the like.

[0057] (Functional Configurations of Communication Terminal 10) Next, the functional configurations of the communication terminal 10 (communication terminal 10A, communication terminal 10B, communication terminal 10C) will be described in detail.

[0058] The communication unit 11 transmits and receives various data (or information) to and from other communication terminals, devices, or systems via the communication network 2. The communication unit 11 starts receiving each state information indicating the state of each communication terminal as a destination candidate from the communication management system 50 before starting communication with a desired destination terminal. Note that this state information indicates not only the operating state (online or offline state) of each communication terminal 10, but also detailed states such as whether communication is possible or in progress when online.

[0059] The operation input reception unit 12 receives various inputs to the communication terminal by the operator. For example, when the operator performs an operation to turn on the power of the communication terminal 10, the operation input reception unit 12 receives the operation and controls the power to be turned on.

[0060] The communication control unit 13, for example, triggered by receiving the above power-on, automatically transmits a login request indicating a request to log in to the communication management system 50 via the communication network 2 from the communication unit 11, and the current IP address of the requesting terminal. Also, when the operator performs an operation to turn off the power of the communication terminal 10, after the communication unit 11 transmits status information indicating that the power is turned off to the communication management system 50, the operation input reception unit 12 turns off the power. Thereby, on the communication management system 50 side, it can be grasped that the communication terminal 10 has changed from power-on to power-off.

[0061] Also, the communication control unit 13 performs various communication controls such as establishing and disconnecting a session for transmitting and receiving content data with another communication terminal 10 via the relay device 30. Note that the communication control unit 13 according to the present embodiment includes the communication ID (Identification) of the communication terminal 10 in the session control information (for example, start request, start response, etc. described later) transmitted to the communication management system 50 and transmits it.

[0062] The communication ID is an example of identification information of an account that can participate in a session for transmitting and receiving content data using the communication terminal 10. The communication ID may be, for example, a user ID that is the identification information of the operator, an app ID that is the identification information of the app, a contract ID that is the identification information of the subscriber of the communication terminal 10, etc. Also, as the communication ID, information in which at least two of characters, numbers, symbols, and various marks are combined can be mentioned. It may be an email address or the like.

[0063] The imaging unit 14 converts the imaging data obtained by imaging the subject into predetermined image (video) data and outputs it. There is one imaging unit on the front side of the communication terminal (imaging the front video) and one on the back side (imaging the back video). There may be a greater number of imaging units.

[0064] The voice input unit 15a converts the voice of the operator into a voice signal by the microphone 114, and then converts this voice signal into predetermined voice data and outputs it. The voice output unit 15b converts the voice data into a voice signal and outputs it to the speaker, causing the speaker 115 to output voice.

[0065] The display control unit 16 causes, for example, the image data included in the content data received by the communication terminal 10 to be displayed on the display unit 109, or the display input unit 110 or the like. Further, the display control unit 16 can transmit the information of the destination list received from the communication management system 50 to the display unit 109, and cause the display unit 109 to display the destination list.

[0066] The storage / reading unit 17 performs processes such as storing various data in the storage unit 18 and reading out various data stored in the storage unit 18.

[0067] In the storage unit 18, for example, authentication information such as the communication ID described above and the password corresponding to the communication ID is stored. In the storage unit 18, the image data and voice data received when communicating with the destination terminal are overwritten and stored each time they are received. Among these, the image is displayed on the display unit 109 or the like by the image data before overwriting, and the voice is output from the speaker 115 by the voice data before overwriting.

[0068] Note that since the communication terminals 10C1 and 10C2 are user terminals to be described later, it is assumed that their functions are the same as those of the communication terminal 10A of the user terminal. However, buttons for receiving operation commands are not displayed on the communication terminals 10C1 and 10C2, or even if an operation command is received, the operation is rejected in the communication management system 50 or the communication terminal 10B.

[0069] Subsequently, each functional configuration included in the communication terminal 10B will be described.

[0070] The operation instruction receiving unit 19a receives operation instruction information for requesting control of a device (mobile device 20) from the communication terminal 10A to the communication terminal 10B via the communication unit 11. This operation instruction information includes, for example, the communication ID of the communication terminal 10 that transmitted the operation instruction information, an operation command indicating the control content to be requested, and the like. There may also be a role to be described later included.

[0071] The device control unit 19c controls the mobile device 20 based on the operation command included in the operation instruction information received by the operation instruction receiving unit 19a. The inter-device communication unit 19d communicates with the mobile device 20 using the short-range wireless communication unit 118.

[0072] <<Functional Configuration of Mobile Device>> The mobile device 20 has, for example, an inter-device communication unit 21, a travel control unit 22, and the like. The inter-device communication unit 21 is realized, for example, by an instruction from the CPU 401 shown in FIG. 5, the external device I / F 404, or the short-range wireless communication unit 405. Here, it is assumed that the inter-device communication unit 21 communicates with the communication terminal 10B using the short-range wireless communication unit 405.

[0073] The travel control unit 22 controls the movement (travel) of the mobile device 20 such as forward movement, backward movement, left turn, right turn, etc. by controlling, for example, the wheel drive unit 406 and the steering unit 407 in FIG. 5 according to the control content acquired from the communication terminal 10A.

[0074] <<Functional Configuration of Communication Management System>> The communication management system 50 has a communication unit 51, a terminal authentication unit 52, a terminal management unit 53, a terminal extraction unit 54, a session management unit 55, a determination unit 56, a storage / reading unit 57, and the like. Each of these units is a function or means realized by operating according to an instruction from the CPU 501 in accordance with a program for the communication management system 50 developed from the HD 504 onto the RAM 503 by any of the components shown in FIG. 4. Also, the communication management system 50 has a storage unit 5000 realized by the HD 504 and the like shown in FIG. 4.

[0075] (Functional Components of the Communication Management System) Next, the functional components of the communication management system 50 will be described in detail. The communication unit 51 transmits and receives various data (or information) with other communication terminals, devices, or systems via the communication network 2.

[0076] The terminal authentication unit 52 authenticates the communication terminal 10 by determining whether the combination of the communication ID and password included in the login request received via the communication unit 51 is included in the authentication management DB (Database) 5002.

[0077] The terminal management unit 53 stores and manages in the terminal management DB 5003 by associating the destination name, operating status, reception date and time of requests, etc., and the IP address of the requesting terminal, etc. for each communication ID. For example, when the operator turns off the power of the communication terminal 10 from ON to OFF, the terminal management unit 53 changes the operating status in the terminal management DB 5003 from online to offline based on the status information indicating that the power is to be turned off sent from the communication terminal 10.

[0078] The terminal extraction unit 54 searches the destination list management DB 5004 using the communication ID of the requesting terminal that has made a login request as a key, and extracts the communication IDs of the destination terminals that can communicate with the requesting terminal. Furthermore, the terminal extraction unit 54 searches the destination list management DB 5004 and also extracts the communication IDs of other communication terminals that have registered the communication ID of the above-mentioned requesting terminal as candidates for the destination terminal.

[0079] Furthermore, the terminal extraction unit 54 searches the above-mentioned terminal management DB 5003 using the communication ID of the extracted destination terminal candidate as a search key, and reads the operating status for each extracted communication ID. As a result, the terminal management unit 53 can obtain the operating status of the destination terminal candidates that can communicate with the requesting terminal that has made a login request. Also, the terminal management unit 53 searches the above-mentioned terminal management DB 5003 using the communication ID of the requester as a search key, and also obtains the operating status of the requesting terminal that has made a login request.

[0080] The session management unit 55 controls the sessions managed by the communication management system 50. The control of the sessions includes, for example, control for establishing a session, control for allowing the communication terminal 10 to participate in the established session, control for disconnecting the session, generation of a session ID, and the like. Further, the session management unit 55 stores and manages, in the session management DB 5005, the communication ID of the communication terminal 10 that requested the start of the session, the communication ID of the destination terminal, etc., in association with the session ID which is the identification information of the session.

[0081] The determination unit 56 makes various determinations. In this embodiment, it determines whether the role of the communication terminal 10 that transmitted the operation command is an operation terminal. Also, it determines the telepresence robot that is the destination of the operation command based on the terminal type.

[0082] The storage / reading unit 57 is realized by instructions from the CPU 501 shown in FIG. 4 and the HDD 505, or by instructions from the CPU 501. The storage / reading unit 57 stores various data in the storage unit 5000 and reads out various data from the storage unit 5000.

[0083] <Example of information managed by the communication management system> Each management DB stored in the storage unit 5000 of the communication management system 50 will be described.

[0084]

Table 1

[0085]

Table 2

[0086]

Table 3

[0087] According to such a configuration, the requesting terminal (for example, communication ID = 1) can start communication only with the registered candidates for the destination terminal (for example, communication IDs = 2, 3, 4, 5). This destination terminal (for example, communication ID = 2) also cannot communicate with the requesting terminal unless it is registered in the destination list management table 701 with the requesting terminal (for example, communication ID = 1) as the destination terminal. Such a mechanism is preferable in that it can reduce the possibility of unexpected communication between communication terminals 10. However, it may be possible to allow communication between communication terminals 10 arbitrarily without requiring registration in the destination list management table 701.

[0088]

Table 4

[0089] <Communication Procedure> Using FIG. 7, the process by which communication terminal 10 starts communication will be described. FIG. 7 is an example of a sequence diagram showing the procedure by which each communication terminal 10 starts communication. Although the mobile device 20 is omitted in FIG. 7, it is integrated with communication terminal 10B.

[0090] S101: Due to the power-on of communication terminal 10B or the operation of the in-company assistant 842, the communication unit 11 of communication terminal 10B designates a communication ID (=1) and a password and sends a login request to the communication management system 50. At this time, communication terminal 10B also sends the fact that it is a "telepresence robot" (its own terminal type). The operating state (online), IP address, and terminal type of communication terminal 10B are registered in the terminal management DB5003.

[0091] S102: Due to the power-on of communication terminal 10C1 or the operation of the home assistant 841, the communication unit 11 of communication terminal 10C1 designates a communication ID (=2) and a password and sends a login request to the communication management system 50. At this time, communication terminal 10B also sends the fact that it is a "user terminal" (its own terminal type). The operating state (online), IP address, and terminal type of communication terminal 10B are registered in the terminal management DB5003.

[0092] S103: Communication terminal 10C1 has communication terminal 10B as a candidate for the destination terminal in the destination list management table. Therefore, the communication unit 51 of the communication management system 50 sends the operating state and terminal type of communication terminal 10B to communication terminal 10C1. The communication unit 11 of communication terminal 10C1 receives the operating state and terminal type of communication terminal 10B.

[0093] S104: The in-home caregiver 841 checks the display indicating that it is communicable with the communication terminal 10B mounted on the telepresence robot, and inputs an operation to start communication with the communication terminal 10B to the communication terminal 10C1. The operation input reception unit 12 of the communication terminal 10C1 receives the operation. Then, the communication unit 11 of the communication terminal 10C1 transmits a session start request to the communication terminal 10B and its own role to the communication management system 50. This role may be fixed to the communication terminal 10C1 or input by the in-home caregiver 841. The telepresence robot or the communication management system 50 only receives operation commands from the operation terminal. Also, when there are a plurality of communication terminals 10 that have declared the operation terminal, the telepresence robot or the communication management system 50 receives operation commands only from the communication terminal 10 that declared first as an example.

[0094] S105,S106: The communication unit 51 of the communication management system 50 receives the session start request and the role, and the session management unit 55 connects the session between communication IDs = 1, 2. That is, it issues a session ID and registers the session ID, relay device ID, communication ID, etc. in the session management DB5005.

[0095] S107: The communication unit 51 of the communication management system 50 requests the communication terminal 10B to connect to the session communicating with the communication terminal 10C1.

[0096] S108, S109: When the in-facility caregiver 842 inputs approval, or when the communication terminal 10B automatically transmits its own communication ID and requests connection of the session, a session is connected between the communication terminals 10B and 10C1. Since the IP addresses of the communication terminals 10B and 10C1 are already registered in the terminal management table, the relay device 30 can transmit and receive content data if it can identify the communication IDs participating in the session.

[0097] The relay device 30 starts transmitting and receiving image data and audio data between the communication terminal 10B (telepresence robot) and the communication terminal 10C1.

[0098] S110: When the power of the communication terminal 10A is turned on or by the operation of the home caregiver 841 or the operator 810, the communication unit 11 of the communication terminal 10A designates the communication ID (=3) and the password and sends a login request to the communication management system 50. At this time, the communication terminal 10A also sends the fact that it is a "user terminal" (its own terminal type). The operating state (online), IP address, and terminal type of the communication terminal 10A are registered in the terminal management DB5003.

[0099] S111: The communication terminal 10A has the communication terminal 10B as a candidate for the destination terminal in the destination list management table. Therefore, the communication unit 51 of the communication management system 50 sends the operating state and terminal type of the communication terminal 10B to the communication terminal 10A. The communication unit 11 of the communication terminal 10A receives the operating state and terminal type of the communication terminal 10B. Actually, the operating states and terminal types of all candidate destination terminals in the destination list table are sent.

[0100] S112: The home caregiver 841 or the operator 810 confirms the display that they can communicate with the communication terminal 10B mounted on the telepresence robot and inputs an operation to start communication with the communication terminal 10B to the communication terminal 10A. The operation input reception unit 12 of the communication terminal 10A receives the operation. Then, the communication unit 11 of the communication terminal 10A sends a session start request and a role to the communication management system 50 for the communication terminal 10B. The role of the communication terminal 10A is an "operation terminal". This role can be fixed to the communication terminal 10A or input by the home caregiver 841 or the operator 810.

[0101] S113 - S115: The communication unit 51 of the communication management system 50 receives the session start request and the role. The session management unit 55 registers in the session management DB5005 by associating the communication ID = 3 with the session to which the communication terminal 10B (telepresence robot) is connected. For example, it is registered in association with the same session ID.

[0102] The relay device 30 starts transmitting and receiving image data and audio data between the communication terminal 10B (telepresence robot), the communication terminal 10C1, and the communication terminal 10A.

[0103] S116: Assume that the home caregiver or operator 810 inputs an operation command to move the telepresence robot to the communication terminal 10A. The operation input reception unit 12 of the communication terminal 10A receives the operation. The communication unit 11 of the communication terminal 10A specifies its own communication ID and transmits the operation command to the communication management system 50.

[0104] S117: The communication unit 51 of the communication management system 50 receives the communication ID and operation command of the communication terminal 10A. The determination unit 56 of the communication management system 50 specifies the session ID from the session management table using the communication ID, and specifies the communication terminal 10B whose terminal type is a telepresence robot from the terminal management table. The communication unit 51 obtains the operation command and the role (operation terminal) of the communication terminal 10A that is the transmission source of the operation command from the terminal management table and transmits them to the communication terminal 10B whose terminal type is a telepresence robot. Therefore, the determination unit 56 gives the operation right to the communication terminal 10A according to the role (operation terminal) of the communication terminal 10A that is the transmission source of the operation command.

[0105] S118: The communication unit 11 of the communication terminal 10B receives the operation command and the role. Since the operation command is received from the communication terminal 10A of the operation terminal, the device control unit 19c transmits the operation command to the moving device 20. That is, the communication terminal 10B that has received the operation command and the role accepts the operation according to the role. When the operation command is received from the communication terminal 10C that is a non-operation terminal, the device control unit 19c discards the operation command without transmitting it to the moving device 20.

[0106] Note that instead of the communication management system 50 transmitting the role to the communication terminal 10A, the determination unit 56 may determine the role and discard the operation command from the communication terminal 10 that is not an operation terminal within the communication management system 50. In this case, since the role and the operation command are not transmitted to the communication terminal 10B, the communication load can be reduced.

[0107] S119: When the power of the communication terminal 10C2 is turned on or an operation is performed by the in-company assistant 842, the communication unit 11 of the communication terminal 10C1 designates a communication ID (=4) and a password and sends a login request to the communication management system 50. At this time, the communication terminal 10C1 also sends the fact that it is a "user terminal" (its own terminal type). The operating state (online), IP address, and terminal type of the communication terminal 10C1 are registered in the terminal management DB5003.

[0108] S120: The communication terminal 10C2 has the communication terminal 10B as a candidate for the destination terminal in the destination list management table. Therefore, the communication unit 51 of the communication management system 50 sends the operating state and terminal type of the communication terminal 10B to the communication terminal 10C2. The communication unit 11 of the communication terminal 10C2 receives the operating state and terminal type of the communication terminal 10B.

[0109] S121: The in-company assistant 842 confirms the display indicating that communication with the communication terminal 10B mounted on the telepresence robot is possible, and inputs an operation to start communication with the communication terminal 10B to the communication terminal 10C2. The operation input reception unit 12 of the communication terminal 10C2 receives the operation. Then, the communication unit 11 of the communication terminal 10C2 sends a session start request and a role to the communication management system 50 for the communication terminal 10B. The role of the communication terminal 10C2 is a "non-operating terminal". This role may be fixed for the communication terminal 10C2 or input by the in-company assistant 842.

[0110] S122~S124: The communication unit 51 of the communication management system 50 receives the session start request and the role. The session management unit 55 registers the communication ID = 4 in the session management DB5005 in association with the session to which the communication terminal 10B is connected. For example, it is registered in association with the same session ID.

[0111] The relay device 30 starts transmitting and receiving image data and audio data among the communication terminal 10B (telepresence robot), the communication terminal 10C1, the communication terminal 10A, and the communication terminal 10C2.

[0112] In this way, in the communication management system 50, the communication terminals 10 of the telepresence robot, the operator, the in-company caregiver, and the home caregiver are connected. Since each communication terminal 10 is connected while distinguishing which role it is, caregivers can communicate with each other within the communication system 1. Since the role of the communication terminal 10C of the caregiver is a non-operating terminal, it cannot send an operation command to the communication terminal 10B and does not interfere with the operation by the communication terminal 10A.

[0113] Note that even if the order of login and participation described in FIG. 7 is swapped, a session can be formed in the same way. Preferably, there is one telepresence robot and one operating terminal each, but such a limitation can be realized by the communication management system 50 excluding in the connection order according to the terminal type and role of the communication terminal 10. Also, the telepresence robot and the operating terminal do not have to be one each. Any number of non-operating terminals can participate.

[0114] <<Information Transmitted by the Communication Terminal at Login>> FIG. 8 is an example of data transmitted by the communication terminal 10 at login, which was described in the process of FIG. 7. As shown in FIG. 8, at the time of login in steps S101, S102, S110, and S119, the communication terminal 10 transmits the communication ID, password, and terminal type to the communication management system 50.

[0115] <<Information Transmitted by the Communication Management System to the Communication Terminal at Login Success>> FIG. 9 is an example of data transmitted by the communication management system 50 to the communication terminal 10 at login success, which was described in the process of FIG. 7. At the time of login success in steps S103, S111, and S120, the operating statuses, etc. of all the communication terminals 10 registered as destination candidates in the destination management table are transmitted to the logged-in communication terminal 10. · Communication ID: Identification information for specifying an account that can use the conference system. · Terminal type: There are types of telepresence robots or user terminals. · Operating status: In addition to the classifications of online, offline, and in communication, in the case of a telepresence robot, there can be two states: "with an operation terminal" and "without an operation terminal". Also, in the case of a user terminal, there can be roles of "operation terminal" and "non-operation terminal". · Session ID: When in communication, an ID that can identify which session is being participated in is transmitted.

[0116] <Examples of Screens for Operation Terminals and Non-Operation Terminals> It is preferable for the operation terminal to display operation buttons and the like, while it is preferably not to display them on the non-operation terminal. Therefore, each communication terminal 10 may switch the display and non-display of the operation buttons according to its own role.

[0117] Figure 10 shows an example of the screen during communication displayed by the communication terminal 10. Figure 10(a) shows an example of the screen of the operation terminal, and Figure 10(b) shows an example of the screen of the non-operation terminal. On the operation terminal, buttons 860 such as cross keys for operating the telepresence robot are displayed in the screen. As examples of operation commands other than operating the telepresence robot, zooming, panning, tilting of the camera, moving to the charging dock, and switching to the mode of following a person may be displayed.

[0118] On the other hand, the non-operation terminal does not display the button 860, and only the image captured by the telepresence is displayed.

[0119] Such control can be realized by the communication terminal 10 storing the role transmitted to the communication management system 50 in the storage unit 18, and the display control unit 16 switching the display and non-display of the button 860 according to the role. When the communication management system 50 creates a screen, it switches whether to display the button 860 according to the role of the communication terminal 10.

[0120] Also, when the button 860 is not displayed on the non-operation terminal, the communication management system 50 or the communication terminal 10B does not need to determine whether to permit an operation.

[0121] <Main Effects> As described above, in the communication system 1 of this embodiment, the communication management system 50 is connected to the communication terminals 10 of the telepresence robot, the operator, the in-company caregiver, and the home caregiver, and each communication terminal 10 is connected while distinguishing which role it is. Therefore, caregivers can communicate with each other within the communication system 1. Since the role of the communication terminal 10C of the caregiver is a non-operating terminal, it cannot send an operation command to the communication terminal 10B and does not interfere with the operation by the communication terminal 10A.

Embodiment

[0122] The telepresence robot is expected to be used for educational purposes for children among the disabled. Especially in Japan, it is expected to be used in extracurricular classes at support schools. In such a use case, in the conventional communication system 1, there has been a demand to make another child experience what a certain child is experiencing. Conventionally, since the operator operates with a mobile terminal such as a tablet (with a small display), it has been difficult for those other than the operator to view the image of the telepresence robot.

[0123] <System Configuration> FIG. 11 is a diagram showing a configuration example of the communication system 1 of this embodiment. In this embodiment, since the components denoted by the same reference numerals in FIG. 2 perform the same functions, there are cases where mainly only the main components of this embodiment will be described.

[0124] The communication system 1 of this embodiment has a communication terminal 10D (an example of a fourth communication terminal) having a large screen of a predetermined size or more (for example, 30 inches or more). The communication terminal 10D can communicate with the communication terminals 10A, 10B, and 10C. Note that the terminal type of the communication terminal 10D is a user terminal and the role is a non-operating terminal. The communication terminal 10D displays the image of the communication terminal 10B on the display. Since the screen of the communication terminal 10D is large, another person can experience the image that the operator is experiencing.

[0125] Note that the processing until connecting the session may be the same as that of the non-operating terminal in the first embodiment. The communication terminal 10D can transmit and receive image data and audio data with the communication terminals 10A, 10B, and 10C. Also, since the communication terminal 10D is a non-operating terminal, it does not display operation buttons.

[0126] In addition to the communication terminal 10D having a large screen, it may project the screen onto a projector. Although the communication terminal 10D has a small screen, it may display the screen on an external large display.

[0127] <Main effects> According to this embodiment, since the communication terminal 10D has a large screen, the content experienced by a certain child can be made to be experienced by another child.

[0128] Note that this embodiment can be similarly applied not only to disabled people or students but also to cases where the communication system 1 is used in a facility tour, etc., when there is a guiding side and a guided side.

[0129] <Other application examples> As described above, the best mode for carrying out the present invention has been described using examples. However, the present invention is not limited to such examples, and various modifications and substitutions can be made without departing from the gist of the present invention.

[0130] For example, in this embodiment, the mobile device 20 traveling on land has been described as an example. However, the mobile device 20 may fly in the air like a drone. Also, it may sail on the sea or move underwater. Also, it may move underground, in narrow roads, or underground. Also, when moving on land, it may move on wheels, perform multi-legged movement such as two legs, three legs, or four legs, or move on a caterpillar (registered trademark).

[0131] Also, not limited to a telepresence robot, the display method of this embodiment may be applied to the video of an imaging device mounted on a device moved by a human.

[0132] In addition, in this embodiment, an example in which one operator operates one mobile device 20 has been mainly described, but it is also possible for one operator to operate a plurality of mobile devices 20. In this case, the communication terminal 10A establishes sessions with a plurality of communication terminals 10B. Conversely, it is also possible for a plurality of operators to operate one mobile device 20. In this case, for example, the operation instruction information last transmitted to the communication terminal 10B may be valid, or a concept of operation authority may be introduced, and only the communication terminal 10A having the operation authority may be allowed to transmit operation instructions. The operation authority can be transferred by an operation such as that of an operator.

[0133] In addition, in this embodiment, it has been described that the operation instruction information is transmitted in the communication session, but it may be transmitted in the session of the content data.

[0134] In addition, in this embodiment, the communication terminal 10 communicates via the relay device 30, but the communication terminal 10 may communicate without passing through the relay device. For example, WebRTC (Web Real-Time Communication) is known as such a communication protocol.

[0135] In addition, the configuration example such as in FIG. 6 is divided according to the main functions in order to facilitate the understanding of the processing by the communication management system 50 and the communication terminal 10. The present invention is not limited by the way of division of the processing units and their names. The processing of the communication management system 50 and the communication terminal 10 can be further divided into more processing units according to the processing content. Also, it can be divided so that one processing unit includes more processing.

[0136] In addition, in this embodiment, for convenience of explanation, the communication management system 50 and the relay device 30 have been described as separate devices, but a device in which the functions of both are integrated may provide the functions of the communication management system 50 and the relay device 30.

[0137] In addition, the communication system 1 may have a plurality of communication management systems 50, and the functions of the communication management system 50 may be distributed and installed in a plurality of servers.

[0138] Further, one or more of the databases that the communication management system 50 has in the storage unit 5000 may be present on the communication network 2.

[0139] Also, the device group described in the embodiments merely shows one of a plurality of computing environments for implementing the embodiments disclosed in this specification. In one embodiment, the communication management system 50 includes a plurality of computing devices such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link including a network and a shared memory, and implement the processes disclosed in this specification.

[0140] Each function of the embodiments described above can be realized by one or more processing circuits. Here, the "processing circuit" in this specification includes a processor programmed to execute each function by software like a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit) designed to execute each function described above, a DSP (digital signal processor), an FPGA (field programmable gate array), and devices such as conventional circuit modules.

Description of Reference Numerals

[0141] 1 Communication system 10 Communication terminal 20 Mobile device 30 Relay device 50 Communication management system

Prior Art Documents

Patent Documents

[0142]

Patent Document 1

Claims

1. A communication system in which a first communication terminal mounted on a mobile device, a second communication terminal that operates the mobile device, and a plurality of third communication terminals communicate with each other via a communication management system, the first communication terminal, the second communication terminal, and the plurality of third communication terminals each have a communication unit configured to transmit a role of the terminal to the communication management system; the communication management system determines whether to give an operation right of the mobile device only to a communication terminal of a predetermined role, or whether the first communication terminal that has received an operation command and the role accepts an operation according to the role; the first communication terminal has an imaging unit, and an image captured by the imaging unit is transmitted from the first communication terminal to the second communication terminal and the third communication terminal; A communication system characterized in that the first communication terminal transmits at least one image of a front image and a rear image captured by the imaging unit to the second communication terminal and a third communication terminal.

2. the role is an operation terminal that operates the mobile device or a non-operation terminal that does not operate the mobile device, The communication system of claim 1, characterized in that the communication management system grants operation rights of the mobile device only to a communication terminal whose role is an operating terminal, or the first communication terminal accepts the operation when an operation command and the role of an operating terminal are received.

3. the communication unit transmits terminal types of the first communication terminal, the second communication terminal, and the plurality of third communication terminals to the communication management system; 3. The communication system according to claim 1, wherein the communication management system transmits an operation command received from the second communication terminal to a communication terminal of a predetermined terminal type.

4. The terminal type is information indicating whether the terminal is installed in the mobile device, 4. The communication system according to claim 3, wherein the communication management system transmits the operation command received from a communication terminal to a communication terminal indicating that the terminal type is installed in the mobile device.

5. the first communication terminal and the third communication terminal have the role of the non-operated terminal; the second communication terminal has the role of the operation terminal, the first communication terminal and the third communication terminal do not display a button for accepting an operation of the mobile device; 3. The communication system according to claim 2, wherein the second communication terminal displays a button for accepting an operation of the mobile device.

6. a fourth communication terminal having a screen of a predetermined size or larger, the fourth communication terminal having the role of a non-operating terminal that does not operate the mobile device; 2. The communication system according to claim 1, wherein the first communication terminal transmits the image to the fourth communication terminal.

7. A communication method for a communication system in which a first communication terminal mounted on a mobile device, a second communication terminal that operates the mobile device, and a plurality of third communication terminals communicate with each other via a communication management system, comprising: a step in which a communication unit of each of the first communication terminal, the second communication terminal, and the plurality of third communication terminals transmits a role of the respective device to the communication management system; a step of determining whether the communication management system gives an operation right of the mobile device only to a communication terminal of a predetermined role, or whether the first communication terminal that has received an operation command and the role accepts an operation according to the role; a step of transmitting an image captured by an imaging unit included in the first communication terminal to the second communication terminal and the third communication terminal; a step of transmitting at least one image of a front image and a rear image captured by the imaging unit from the first communication terminal to the second communication terminal and a third communication terminal; A communication method comprising:

8. A communication management system that communicates with a first communication terminal mounted on a mobile device, a second communication terminal that operates the mobile device, and a plurality of third communication terminals, a communication unit that receives a role of the own device from the first communication terminal, the second communication terminal, and the plurality of third communication terminals; a determination unit that grants an operation right of the mobile device only to a communication terminal having a predetermined role; Function as a the first communication terminal has an imaging unit, and an image captured by the imaging unit is transmitted from the first communication terminal to the second communication terminal and the third communication terminal; A program in which the first communication terminal transmits at least one of a front image and a rear image captured by the imaging unit to the second communication terminal and a third communication terminal.

Images