COMMUNICATION TERMINAL, COMMUNICATION SYSTEM, SHARING METHOD, AND PROGRAM

The communication terminal enables mode changes to restrict input to specific areas during remote video calls, addressing the challenge of unintended input and reducing discussion times.

JP7577944B2Active Publication Date: 2024-11-06RICOH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2020125345
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-07-22
Publication Date
2024-11-06
Estimated Expiration
2040-07-22

AI Technical Summary

Technical Problem

In remote video calls, it is challenging to accurately convey instructions for inputting information, leading to longer discussion times as customers may input in unintended areas, complicating the process.

Method used

A communication terminal that allows input information to be shared between terminals, enabling a mode change to restrict input to specific areas, using icons to designate input-enabled zones and disabling input elsewhere, facilitated by a mode change instruction received from another terminal.

Benefits of technology

This solution reduces discussion time by ensuring accurate input in designated areas, preventing unauthorized input and streamlining the communication process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577944000001
    Figure 0007577944000001
  • Figure 0007577944000002
    Figure 0007577944000002
  • Figure 0007577944000003
    Figure 0007577944000003
Patent Text Reader

Abstract

To solve the problem in which: in recent years, from the viewpoint of streamlining and efficiency of store business, a remote concierge service (remote customer service) has come into practice; it is possible to display images (screens) of the same application, etc. on an explainer terminal used by an explainer and a store terminal used by a customer and allow the customer to input their name, etc. in a predetermined entry column with an electronic pen, etc. in accordance with an instruction from the explainer through a video call; however, in the remote video call, the explainer may have difficulty conveying the details compared with a case where the explainer instructs the customer face-to-face; therefore, when the explainer inputs the name, etc. in an unintended place, the customer may have a long conversation.SOLUTION: A customer terminal changes, based on an instruction from an explainer terminal, from an input enabled mode in which input to the entirety of a displayed predetermined area is enabled to an input disabled mode in which input to the entirety of the predetermined area is disabled, and enables input using an electronic pen 10 performed by a customer to a specific area 161 of the predetermined area.SELECTED DRAWING: Figure 22
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a communication terminal, a communication system, a sharing method, and a program. [Background technology]

[0002] In recent years, remote concierge (remote customer service) services have been introduced from the viewpoint of streamlining and streamlining store operations. As a result, even if a customer has to wait his turn to be served because a store employee is busy serving customers, the customer can receive customer service by accessing a terminal of a person (concierge) with specialized skills at a remote service center from a store terminal via the Internet or the like and having a video call with the person (see Patent Document 1).

[0003] Recently, in addition to manned stores that have employees who serve customers, unmanned stores that have no employees have started to be operated. These unmanned stores also have store terminals installed, and customers can visit the unmanned store and receive customer service by using the store terminal to have a video call with an instructor at the service center.

[0004] Furthermore, it is also possible to display the same image (screen) of an application form or the like on the presenter terminal and the store terminal, and have the customer enter their name or the like in a prescribed entry field with an electronic pen or the like in response to instructions from the presenter via video call. Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the case of a remote video call, it can be difficult for the presenter to convey information compared to when the presenter gives instructions to the customer face-to-face. Therefore, there is a problem that the customer may enter their name or other information in a place that the presenter does not intend, which may result in the discussion taking longer. [Means for solving the problem]

[0006] The invention according to claim 1 is a communication terminal that shares input information with other communication terminals, and changes the mode from a first mode in which input to the entirety of a displayed predetermined area is possible to a second mode in which input to the entirety of the predetermined area is impossible. Change the mode A mode changing means; A receiving means for receiving a mode change instruction transmitted from the other communication terminal; and an input enabling means for enabling input to a specific area of ​​the predetermined area in the second mode. the instruction to change mode includes information on an inputtable candidate area which is a candidate for the specific area, the mode change means changes the mode from the first mode to the second mode based on the instruction to change mode received by the receiving means, the receiving means receives change information transmitted by the other communication terminal pressing an icon corresponding to the inputtable candidate area, and the input enabling means enables input to the inputtable candidate area corresponding to the icon pressed by the other communication terminal based on the change information received by the receiving means. The present invention relates to a communication terminal. Effect of the Invention

[0007] As described above, according to the present invention, it is possible to prevent the explanation time from becoming too long. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a conceptual diagram showing a communication route according to the embodiment; [Diagram 2] FIG. 13 is an image diagram showing a situation in which a customer at a store and an instructor at a service center are remotely making a video call. [Diagram 3] FIG. 2 is a hardware configuration diagram of each terminal and each device that constitutes the communication system of the present embodiment. [Figure 4] FIG. 2 is a functional block diagram of the communication system according to the embodiment. [Diagram 5] 13 is a conceptual diagram showing a common stroke data management table managed by a store terminal, a communication management device, and a presenter terminal. FIG. [Figure 6] FIG. 1A is a conceptual diagram showing an authentication management table, and FIG. 1B is a conceptual diagram showing a terminal state management table. [Figure 7] 1A is a conceptual diagram showing a destination list management table, FIG. 1B is a conceptual diagram showing a session management table, and FIG. 1C is a conceptual diagram showing a communication information management table. [Figure 8] FIG. 2 is a diagram illustrating the configuration of each image layer. [Figure 9]FIG. 11 is a sequence diagram showing a preparation process for a video call of a store terminal, and in particular, a process for acquiring the operating status of each terminal. [Figure 10] FIG. 13 is a diagram showing a screen displayed on a store terminal. [Figure 11] FIG. 11 is a sequence diagram showing detailed processing of communication establishment control. [Figure 12] FIG. 11 is a sequence diagram showing a process of activating a bidirectional input function. [Figure 13] 13 is a diagram showing a display example of an input image on the presenter terminal side. FIG. [Figure 14] FIG. 11 is a sequence diagram showing a process for sharing a document image. [Figure 15] FIG. 1A is a diagram showing an image displayed on the presenter terminal side, and FIG. 1B is a diagram showing an image displayed on the store terminal side. [Figure 16] FIG. 11 is a sequence diagram showing a sharing process of an inputtable candidate area. [Figure 17] FIG. 1A is a diagram showing an image displayed on the presenter terminal side, and FIG. 1B is a diagram showing an image displayed on the store terminal side. [Figure 18] FIG. 1A is a diagram showing an image displayed on the presenter terminal side, and FIG. 1B is a diagram showing an image displayed on the store terminal side. [Figure 19] FIG. 11 is a sequence diagram showing a sharing process of input permission. [Figure 20] FIG. 1A is a diagram showing an image displayed on the presenter terminal side, and FIG. 1B is a diagram showing an image displayed on the store terminal side. [Figure 21] FIG. 11 is a sequence diagram showing a sharing process of input contents. [Figure 22] FIG. 1A is a diagram showing an image displayed on the presenter terminal side, and FIG. 1B is a diagram showing an image displayed on the store terminal side. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Overall configuration of communication system] The overall configuration of the communication system will be described with reference to Fig. 1. Fig. 1 is an overall configuration diagram of a communication system according to an embodiment of the present invention. Here, a customer, a travel agency that sells travel products to the customer, and a system provider that provides a system for remote concierge (remote customer service) to the travel agency appear.

[0010] A travel agency operates staffed stores, unmanned stores, and service centers in each of several areas, such as Kanto and Kansai. Although only one unmanned store US is shown in FIG. 1, there may be multiple unmanned stores, and the store may be staffed.

[0011] In a manned store, an employee usually introduces travel products to customers who visit the store, but there are cases where the employee is busy serving other customers and cannot serve new customers. Also, in an unmanned store, there are no employees to serve customers in the first place. In such a case, a customer can use the store terminal 1 in a manned store or unmanned store US to access the presenter terminal 9 of any presenter E in the service center SC via a communication network 100 such as the Internet, and receive remote customer service regarding travel products from the presenter. In the example of FIG. 1, customer C is a person who has visited the unmanned store US.

[0012] The store terminal 1 and the presenter terminal 9 are configured by a PC (personal computer). A plurality of store terminals may be installed in a staffed store or an unstaffed store US. Also, a plurality of presenter terminals may be installed in a service center SC.

[0013] The relay device 3 and the communication management device 5 are computers such as a server. The relay device 3 relays image data and audio data when a video call is made between any of the store terminals 1 and any of the presenter terminals 9. The communication management device 5 centrally manages login authentication from the store terminal 1 and the presenter terminal 9, management of the communication status of the store terminal 1 and the presenter terminal 9, management of the destination list, and communication status of the relay device 3. The relay device 3 and the communication management device 5 will be described in detail later.

[0014] The store terminal 1, the presenter terminal 9, the relay device 3, and the communication management device 5 can be connected to a communication network 100 such as the Internet.

[0015] The relay device 3 and the communication management device 5 may be constructed by a single computer, or may be constructed by a plurality of computers to which each unit (function or means) is divided and assigned arbitrarily.

[0016] Fig. 2 is an image diagram showing a situation in which a customer at a store and a presenter at a service center are having a remote video call. Fig. 2 shows a situation in which a customer C who has visited an unmanned store US is using a store terminal 1, and a presenter E at the service center SC is using a presenter terminal 9. In this way, customer C can receive an explanation of travel products and the like from presenter E through remote customer service. Note that a "Start two-way input" button 990 is displayed on the presenter terminal 9.

[0017] [Hardware configuration of communication system] Next, using FIG. 3, the hardware configuration of each terminal (store terminal 1, presenter terminal 9, and each device (relay device 3, communication management device 5, service information management device 7) that constitutes the communication system shown in FIG. 1 will be described. Note that in this embodiment, it is assumed that each terminal and each device have the same configuration, so the hardware configuration of the store terminal 1 will be described and descriptions of the hardware configurations of the other terminals and devices will be omitted.

[0018] As shown in FIG. 3, the store terminal 1 is composed of a computer and has a CPU 101, ROM 102, RAM 103, HD 104, HDD (Hard Disk Drive) controller 105, display 106, external device connection I / F (Interface) 108, network I / F 109, bus line 110, keyboard 111, pointing device 112, DVD-RW (Digital Versatile Disk Rewritable) drive 114, and media I / F 116.

[0019] Among these, the CPU 101 controls the operation of the entire computer. The ROM 102 stores programs such as IPL used to drive the CPU 101. The RAM 103 is used as a work area for the CPU 101. The HD 104 stores various data such as programs. The HDD controller 105 controls reading and writing of various data from and to the HD 104 under the control of the CPU 101. The display 106 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 108 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 109 is an interface for data communication using the communication network 100. The bus line 110 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 101 shown in FIG. 3.

[0020] Moreover, the keyboard 111 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 112 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 114 controls reading and writing of various data from a DVD-RW 113 as an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R or a Blu-ray Disc, etc. The media I / F 116 controls reading and writing (storing) of data from a recording medium 115 such as a flash memory.

[0021] Also, a microphone 121 as an example of a sound collection device, a speaker 122 as an example of a sound output device, and a camera 123 as an example of an imaging device are connected to the external device connection I / F. Note that, although the microphone 121, the speaker 122, and the camera 123 are shown as external devices in Fig. 3, they may be built-in devices without going through the external device connection I / F.

[0022] [Functional configuration of the communication system] Next, each functional configuration of the communication system according to the present embodiment will be described with reference to Fig. 3 to Fig. 8. Fig. 4 is a block diagram showing each functional configuration of the communication system according to the present embodiment.

[0023] <Store terminal functional configuration> First, the functional configuration of the store terminal 1 will be described with reference to Fig. 3 and Fig. 4. As shown in Fig. 4, the store terminal 1 has a transmitting / receiving unit 11, a receiving unit 12, a communication control unit 13, a display control unit 14, a determining unit 15, a two-way input unit 16, a mode changing unit 17, a calculating unit 18, and a storage / reading processing unit 19. Each of these units is a function or a means for performing a function that is realized by operating any of the components shown in Fig. 3 in response to an instruction from the CPU 101 in accordance with a program loaded from the HD 104 onto the RAM 103. The store terminal 1 also has a storage unit 1000 constructed by the RAM 103 and the HD 104 shown in Fig. 3.

[0024] (Stroke information management table) FIG. 5 is a conceptual diagram showing a common stroke data management table managed by each of the store terminal, the communication management device, and the presenter terminal.

[0025] In the storage unit 1000, a stroke information management DB 1001 is constructed by a stroke information management table as shown in Fig. 5. In this stroke information management table, stroke information sent from the presenter terminal 9 in steps S248 to S250 described later and stroke information accepted by the store terminal 1 from the customer C in step S281 described later are managed. The stroke information includes each piece of information related to a stroke ID, a connection ID, a screen ID, stroke coordinates, a stroke color, a stroke width, whether or not the area is closed, and whether or not input is possible.

[0026] Among these, the stroke ID is an example of stroke identification information for identifying stroke information. The connection ID is the same as the connection ID in the communication information management table (see FIG. 7(C)). The screen ID is an example of screen identification information for identifying the input screen 100 (900). The stroke coordinates are coordinates for identifying the position of the stroke on the input screen 100 (900). The stroke color is information indicating the color of the stroke set by the user (customer C or presenter E). The stroke width is information indicating the width of the stroke set by the user (customer C or presenter E). The "presence or absence of closed area" is information indicating whether or not a closed area is formed by drawing the inputtable candidate area 940 with the electronic pen 90 as shown in FIG. 17(A), for example. The input possibility is information indicating whether the input possible mode is set or the input impossible mode is set. In the input possible mode, the customer C can input characters, etc., on the document image with the electronic pen 10 at the store terminal 1 side. The input-prohibited mode is information indicating whether the store terminal 1 is set to an input-prohibited mode in which customer C cannot input characters or the like onto the document image using the electronic pen 10. This will be explained in more detail later. The input-possible mode is an example of a first mode. The input-prohibited mode is an example of a second mode. Furthermore, input may be made by directly touching the display 106 (906) with a finger or the like without using the electronic pen 10 (90).

[0027] (Functional configuration of store terminal) Next, we will explain each component of the store terminal 1. The transmission / reception unit 11 is realized by commands from the CPU 101 shown in Fig. 3, the external device connection I / F 108, and the network I / F 109, and transmits and receives various data (or information) to and from other terminals, devices, or systems via the communication network 100.

[0028] The reception unit 12 is realized mainly by commands from the CPU 101 shown in FIG. 3, as well as the keyboard 111, the pointing device 112, the operation buttons 108, and the power switch 109, and receives various inputs from the user.

[0029] The communication control unit 13 is realized by an instruction from the CPU 101 shown in Fig. 3, and inputs image data obtained by an external camera connected to the external device connection I / F 108 by capturing an image of a subject. The communication control unit 13 also inputs sound data obtained by an external microphone connected to the external device connection I / F 108 by collecting sound. The communication control unit 13 also outputs image data received from the other party of the video call and image data obtained by its own terminal to the display control unit 14. The communication control unit 13 also outputs sound data received from the other party of the video call and sound data obtained by its own terminal to an external speaker connected to the external device connection I / F 108.

[0030] The display control unit 14 is mainly realized by instructions from the CPU 101 shown in FIG. 3, and displays an image by outputting image data to the display 106 or an external display connected to the external device connection I / F.

[0031] The determination unit 15 is mainly realized by instructions from the CPU 101 shown in Fig. 3, and performs various determinations, the contents of which will be described later.

[0032] The two-way input unit 16 is mainly realized by commands from the CPU 101 shown in Fig. 3, and is a function for realizing two-way input between the communication terminal 1 and the presenter terminal 9. For example, characters input on the communication terminal 1 side are displayed on the presenter terminal 9 side.

[0033] The mode change section 17 is mainly realized by instructions from the CPU 101 shown in FIG. 3, and changes the mode from the input enabled mode to the input disabled mode and from the input disabled mode to the input enabled mode.

[0034] The calculation unit 18 calculates stroke coordinates based on the strokes input by the customer C for reproducing the same strokes on the presenter terminal 9 on the other side.

[0035] The storage / read processing unit 19 is mainly executed by instructions from the CPU 101 shown in FIG. 3 and the HDD controller 105, and performs processes such as storing various data in the memory unit 1000 and reading out various data stored in the memory unit 1000.

[0036] Furthermore, image data and sound data received during communication with other terminals are overwritten and stored every time they are received in the storage unit 1000. Of these, images are displayed on the display 106 or an external display using the image data before being overwritten, and sound is output from an external speaker using the sound data before being overwritten.

[0037] <Functional configuration of presenter terminal> Next, the functional configuration of the presenter terminal 9 will be described with reference to Fig. 3 and Fig. 4. As shown in Fig. 4, the presenter terminal 9 has a transmitting / receiving unit 91, a receiving unit 92, a communication control unit 93, a display control unit 94, a determining unit 95, a two-way input unit 96, a mode changing unit 97, a calculating unit 98, and a storing / reading processing unit 99. The presenter terminal 9 also has a memory unit 9000 constructed by the RAM 103 and the HD 104 shown in Fig. 3. A stroke information management DB 9001 is constructed in the memory unit 9000. This stroke information management DB 9001 has the same structure as the stroke information management DB 1001 already described, and therefore a description thereof will be omitted.

[0038] The transmitter / receiver 91, receiver 92, communication controller 93, display controller 94, judgement unit 95, two-way input unit 96, mode change unit 97, calculation unit 98, and storage / readout processor 99 of the presenter terminal 9 have the same functions as the transmitter / receiver 11, receiver 12, communication controller 13, display controller 14, judgement unit 15, two-way input unit 16, mode change unit 17, calculation unit 18, and storage / readout processor 19 of the store terminal 1, respectively, and therefore descriptions thereof will be omitted. The calculation unit 98 calculates stroke coordinates for reproducing the same strokes at the other store terminal 1 based on the strokes input by the presenter E.

[0039] <Functional configuration of relay device> Next, the functional configuration of the relay device 3 will be described with reference to Fig. 3 and Fig. 4. As shown in Fig. 4, the relay device 3 has a transmission / reception unit 31 that also functions as a transfer unit, a determination unit 32, and a storage / readout processing unit 39. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 101 in accordance with a program loaded from the HD 104 onto the RAM 103. The relay device 3 also has a storage unit 3000 constructed by the RAM 103 and the HD 104 shown in Fig. 3.

[0040] (Functional configuration of relay device) Next, each functional configuration of the relay device 3 will be described in detail. The transmission / reception unit 31 of the relay device 3 shown in Fig. 4 is realized by commands from the CPU 101 shown in Fig. 3 and the network I / F 109 shown in Fig. 3, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 100. The transmission / reception unit 31 also serves as a transfer unit, and transfers image data and sound data transmitted from a specific terminal to another terminal.

[0041] The determination unit 32 is realized by an instruction from the CPU 101 shown in FIG. 3, and performs a determination of the data delay state and the like.

[0042] The storage / readout processing unit 39 is realized by instructions from the CPU 101 shown in FIG. 3 and the HDD 105 shown in FIG. 3, and performs processing such as storing various data in the memory unit 3000 and reading out various data stored in the memory unit 3000.

[0043] <Functional configuration of the communication management device> Next, the functional configuration of the communication management device 5 will be described with reference to Figures 3 and 4. As shown in Figure 4, the communication management device 5 has a transmission / reception unit 51, a selection unit 53, a creation unit 54, a judgment unit 55, and a storage / readout processing unit 59. Each of these units is a function or a means of functioning that is realized when any of the components shown in Figure 4 operates in response to an instruction from the CPU 101 in accordance with a program loaded from the HD 104 onto the RAM 103. The communication management device 5 also has a storage unit 5000 constructed by the RAM 103 and HD 104 shown in Figure 3.

[0044] (Authentication Management Table) Fig. 6(A) is a conceptual diagram showing an authentication management table. An authentication management DB 5001 configured with an authentication management table as shown in Fig. 6(A) is constructed in the storage unit 5000. In this authentication management table, each password is associated with each communication ID of all terminals 1, 9 managed by the communication management device 5 and managed. Note that the password is an example of authentication information, and the authentication information also includes an access token.

[0045] (Terminal status management table) Fig. 6(B) is a conceptual diagram showing a terminal status management table. In the storage unit 5000, a terminal status management DB 5002 configured by the terminal status management table shown in Fig. 6(B) is constructed. In this terminal status management table, for each communication ID of each terminal 1, 9, the destination name when each terminal 1, 9 is the destination, the operating state of each terminal 1, 9, the reception date and time when login request information described later is received by the communication management device 5, and the IP address of the store terminal 1, 9 are associated and managed. The communication ID, destination name, and terminal IP address are stored when each terminal 1, 9 is pre-registered to receive the service provided by the communication management device 5.

[0046] (Destination list management table) Fig. 7(A) is a conceptual diagram showing a destination list management table. A destination list management DB 5003 configured by a destination list management table as shown in Fig. 7(A) is constructed in the storage unit 5000. In this destination list management table, all communication IDs of destination terminals registered as candidates for destination terminals are associated with the communication ID of a starting terminal requesting the start of a call and are managed. Note that the destination list is an example of destination information, and the destination information may not be in a list format, and may instead list information related to destinations such as communication IDs.

[0047] (Session management table) Fig. 7(B) is a conceptual diagram showing a session management table. A session management DB 5004 configured by a session management table as shown in Fig. 7(B) is constructed in the storage unit 5000. In this session management table, for each communication session ID for identifying a session for performing mutual communication between each terminal and the relay device 3, the relay device ID of the relay device 3 used, the communication ID of the starting terminal, the communication ID of the destination terminal, the reception delay time (ms) when the image data is received at the destination terminal, and the reception date and time when the delay information indicating this delay time is sent from the destination terminal and received by the communication management device 5 are associated and managed.

[0048] (Communication information management table) Fig. 7(C) is a conceptual diagram showing a communication information management table. A communication information management DB 5005 configured by a communication information management table as shown in Fig. 7(C) is constructed in the storage unit 5000. In this communication information management table, for each connection ID for identifying mutual communication performed between each terminal and the relay device 3, the IP address of the relay device of the relay device 3 used and the communication ID of the terminal with which the mutual communication is performed are associated and managed.

[0049] (Stroke information management table) A stroke information management DB 5006 is stored in the storage unit 5000 and is configured by a stroke information management table as shown in Fig. 5. The stroke information management DB 5006 has the same structure as the stroke information management DB 1001 already described, and therefore a description thereof will be omitted.

[0050] (Functional configuration of the communication management device) Next, a detailed description will be given of each functional configuration of the communication management device 5. In the following, in describing each functional configuration of the communication management device 5, the relationship between each component shown in FIG. 3 and the main components for realizing each functional configuration of the communication management device 5 will also be described.

[0051] The transmission / reception unit 31 of the communication management device 5 shown in Figure 4 is realized by instructions from the CPU 101 shown in Figure 3 and the network I / F 109, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 100.

[0052] The selection unit 53 is realized by an instruction from the CPU 101 shown in FIG. 3, and selects the optimum relay device for mutual communication based on the IP address of each terminal.

[0053] The creation unit 54 is realized by an instruction from the CPU 101 shown in Fig. 3, and creates communication information in response to a request from each terminal to start mutual communication. This communication information includes the IP address of the relay device selected by the selection unit 53, the above-mentioned communication ID, etc.

[0054] The determination unit 55 is realized by instructions from the CPU 101 shown in FIG. 3, and performs various determinations.

[0055] The storage / read processing unit 59 is realized by instructions from the CPU 101 and the HDD controller 105 shown in FIG. 3, and performs processing such as storing various data in the memory unit 5000 and reading out various data stored in the memory unit 5000.

[0056] <Image layer> Next, each image layer displayed on the electronic whiteboard 1 will be described with reference to FIG. 8. As shown in FIG. 8, the display control unit 14 (94) creates an image in which four image layers are superimposed. FIG. 8 is a configuration diagram of each image layer. Each image layer is configured as layers of a UI image (A), a stroke image (B1) of the store terminal 1, a stroke image (B2) of the presenter terminal 9, and a document image, from the left, in the order in which the user can see them even when they are superimposed. Among these, the UI image (A), the stroke image (B1) of the store terminal, the stroke image (B2) of the presenter terminal, and the background image (C) are layers of images displayed on the store terminal 1 and the presenter terminal 9. Note that, in addition to these image layers, an image of the other party's base is displayed by a picture-in-picture technology. Each image layer will be described below.

[0057] The layer of the UI image (A) is a layer that displays icons and the like and accepts UI operations such as input or selection so that the user can perform a predetermined input or selection with the electronic pen 10 (90). In Fig. 8, task bars 920 and 930 (130), which will be described later, are displayed.

[0058] The layer of the stroke image (B1) of the store terminal 1 is a layer for sharing the stroke image drawn by the customer with the electronic pen 10 with the other party's presenter terminal 9. In this case, the electronic whiteboard 1 transmits stroke information to the other party's presenter terminal 9 via the communication network 100. Here, a stroke image b1 showing the input contents such as the customer's name is displayed.

[0059] The layer of the stroke image (B2) of the presenter terminal 9 is a layer for sharing the stroke image drawn by the presenter E with the electronic pen 90 with the other party's customer terminal 1. In this case, the presenter terminal 9 transmits stroke information to the other party's customer terminal 1 via the communication network 100. Here, a stroke image b2 showing input contents such as an inputtable area described later is displayed.

[0060] The document image (C) layer is a layer for the presenter E to share the document image with the other party's customer terminal 1. In this case, the presenter terminal 9 transmits data of the document image to the other party's customer terminal 1 via the communication network 100. Here, as an example of the document image, an image of an individual membership application form as shown in FIG. 15 is displayed.

[0061] [Processing or Operation of the Embodiment] Next, the processing or operation in the communication system according to this embodiment will be described with reference to Fig. 9 to Fig. 22. Here, as shown in Fig. 2, a case will be described in which a customer C enters an unmanned store US, starts a video call to a service center SC using a store terminal 1, and receives an explanation from a presenter E on how to write document information. Note that, hereinafter, the store terminal 1 as the request source requesting the start of a video call may be referred to as the "start terminal", and the presenter terminal 9 as the destination (relay destination) to which the request is made may be referred to as the "destination terminal".

[0062] <Video call start process> When customer C performs an operation to start a video call on the store terminal 1, as shown in Fig. 9, the creation unit 54 of the store terminal 1 creates a session ID for identifying the communication session by itself (S22). Next, the transmitting / receiving unit 11 transmits login request information indicating a login request to the communication management device 5 via the communication network 100 (S23). This login request information includes the session ID created in step S22, the communication ID and password of the own terminal. Note that the communication ID here is an ID for identifying the store terminal 1 that transmitted the login request in step S23, and may be a terminal ID for identifying the terminal, or a user ID for identifying a user such as a customer.

[0063] Next, the storage / read processing unit 59 of the communication management device 5 searches the authentication management table (see FIG. 6(A)) using the communication ID and password included in the login request information received via the transmission / reception unit 51 as search keys, and performs terminal authentication by determining whether the same communication ID and the same password are managed (S24). Here, the explanation will continue assuming that the same communication ID and the same password are managed by the storage / read processing unit 59.

[0064] If the storage / readout processing unit 59 determines that the login request is from a terminal with valid usage authority, assuming that the same communication ID and the same password are managed, the storage / readout processing unit 59 changes the operation status field part of the record of the communication ID received in step S23 to "Online (available for calls)" in the terminal status management table (see FIG. 6(B)), and stores the reception date and time when the login request was received in step S23 in the reception date and time field part (S25). As a result, for example, the communication ID "01aa" is managed in the terminal status management table in association with the operation status "Online (available for calls)", the reception date and time "2020.4.10.13:40", and the IP address "1.2.1.4". Note that the IP address of the terminal may not be registered in advance, but may be transmitted from the store terminal 1 in step S23.

[0065] Next, the storage / read processing unit 59 adds a new record including the communication ID and password of the store terminal 1 received in step S23 to the session management table (see FIG. 7(B)) and manages it (S26). Then, the transmitting / receiving unit 51 of the communication management device 5 transmits authentication result information indicating the authentication result obtained by the processing of step 24 to the store terminal 1 that made the login request via the communication network 100 (S27).

[0066] When the transmitting / receiving unit 11 of the login request terminal (store terminal 1) receives authentication result information indicating that the terminal has been determined to have valid usage authority, the transmitting / receiving unit 11 transmits destination list request information indicating a request for a destination list to the communication management device 5 via the communication network 100 (S28). As a result, the transmitting / receiving unit 51 of the communication management device 5 receives the destination list request information.

[0067] Next, the storage / read processing unit 59 searches the destination list management table (see FIG. 7(A)) using the communication ID "01ab" of the login request terminal (store terminal 1) as a search key, reads out the communication IDs of destination candidates that can communicate with the login request terminal (store terminal 1), and reads out the destination names corresponding to these communication IDs from the terminal status management table (see FIG. 6(B)) (step S29). Here, the communication IDs of the destination candidates that correspond to the communication ID "01ab" of the login request terminal (store terminal 1) and the destination names corresponding to these are extracted.

[0068] Next, the storage / read processing unit 59 of the communication management device 5 reads out data of the destination list frame from the storage unit 5000 (S30). Then, the transmission / reception unit 51 transmits "destination list information (destination list frame, communication ID, destination name)" including this destination list frame and the communication ID and destination name read out by the storage / read processing unit 59 to the login request terminal (store terminal 1) (S31). As a result, in the login request terminal (store terminal 1), the transmission / reception unit 11 receives the destination list information, and the storage / read processing unit 19 stores the destination list information in the storage unit 1000 (S32).

[0069] Thus, in this embodiment, rather than each terminal managing the destination list information, the communication management device 5 centrally manages the destination list information of all terminals. Therefore, even if a new terminal is added to the communication system, a new model of terminal is added to replace an existing terminal, or the appearance of the destination list frame needs to be changed, the communication management device 5 handles all of this collectively, eliminating the need to change the destination list information on each terminal side.

[0070] In addition, the storage / read processing unit 59 of the communication management device 5 searches the terminal status management table (see Figure 6 (B)) using the communication ID of the destination candidate read out as a search key, and obtains each operating status of the communication terminal as a destination candidate by reading out the corresponding operating status for each of the communication IDs (S33).

[0071] Next, the transmitter / receiver unit 51 transmits "terminal status information" including each communication ID used as a search key in step S26 above and the operating status of each corresponding destination terminal to the starting terminal (store terminal 1) via the communication network 100 (S34).

[0072] Next, the storage / readout processing unit 19 of the starting terminal (store terminal 1) sequentially stores the terminal status information received from the communication management device 5 in the storage unit 1000 (S35). Thus, by receiving the status information of each of the above terminals, the starting terminal (store terminal 1) can obtain the current operating status of each of the presenter terminals 9 and other terminals that are potential destinations that can communicate with the starting terminal (store terminal 1).

[0073] Next, the display control unit 14 of the login request terminal (store terminal 1) creates a destination list that reflects the status of the terminal as a destination candidate based on the destination list information and terminal status information stored in the storage unit 1000 (S36). This destination list is displayed on the store terminal 1 as shown in FIG.

[0074] Meanwhile, the storage / read processing unit 59 of the communication management device 5 searches the destination list management table (see FIG. 7(A)) based on the communication ID of the login request terminal (store terminal 1) to extract the communication IDs of other terminals that have registered the communication ID of the login request terminal (store terminal 1) as a destination candidate (S37).

[0075] Next, the storage / read processing unit 59 of the communication management device 5 searches the terminal status management table (see FIG. 6(B)) based on the communication ID of the login request terminal (store terminal 1) and obtains the operating status of the login request terminal (store terminal 1) (S38).

[0076] Then, the transmitting / receiving unit 51 transmits "terminal status information" including the communication ID and operating status of the login request terminal (store terminal 1) acquired in the above step S38 to the terminals related to the communication ID extracted in the above step S37, which have an operating status of "Online" in the terminal status management table (see FIG. 6(B)) (S39). When the transmitting / receiving unit 51 transmits the terminal status information to the presenter terminal 9, the transmitting / receiving unit 51 refers to the IP address of the communication terminal managed in the terminal status management table (see FIG. 6(B)) based on each communication ID. This makes it possible to convey the communication ID and operating status of the login request terminal (store terminal 1) to each of the other destination terminals that can communicate with the login request terminal (store terminal 1) as a destination candidate. This makes it possible to display the status of the destination candidate (presenter terminal 9) as shown in FIG. 10(C) (S40).

[0077] <Communication establishment control process> Next, the process of controlling the establishment of communication between the store terminal 1 and the presenter terminal 9 (see S90) will be described in detail with reference to Fig. 11. Fig. 11 is a sequence diagram showing the detailed process of the control of the establishment of communication.

[0078] First, the store terminal 1 transmits start request information indicating a request to start video communication to the communication management device 5 (step S101). This start request information includes the communication ID of the start terminal (here, the store terminal 1) and the communication ID of the presenter E of the destination terminal (here, the presenter terminal 9). In this way, the communication management device 5 receives the start request information.

[0079] Next, in the communication management device 5, the selection unit 53 selects the nearest relay device based on each IP address of the starting terminal and the destination terminal (step S102). The IP address of each terminal is managed in a terminal management table, and the IP address of each relay device is stored in advance in the storage unit 5000. Next, the creation unit 54 creates communication information including the IP address of the relay device and the above-mentioned communication ID (step S103). Then, the storage / readout processing unit 59 manages the communication information management DB 5005 (see FIG. 7(C)) by storing the communication ID and the IP address of the relay device, which are the contents of the communication information, and the communication IDs of the presenters of the starting terminal and the destination terminal received in step S101 in association with each other. Thereafter, the transmission / reception unit 51 transmits start request information to the destination terminal (the presenter terminal 9 in this case) (step S105). This start request information includes the communication information in addition to each communication ID sent from the starting terminal in the above-mentioned step S101. As a result, the transmission / reception unit 91 of the destination terminal receives the start request information.

[0080] Next, in the destination terminal, the determination unit 15 of the presenter terminal 9 determines whether the communication ID of the starting terminal (store terminal 1) received in step S105 is the same as the communication ID of the store terminal 1 previously acquired (S106). In this case, since they are basically the same, the transmission / reception unit 91 transmits response possible information to the communication management device 5 (step S107). The response possible information in this case indicates that a response is possible, and includes the communication ID of the starting terminal. As a result, the transmission / reception unit 51 of the communication management device 5 receives the response possible information.

[0081] Next, the storage / read processing unit 59 of the communication management device 5 searches the communication information management DB 5005 using the communication ID of the starting terminal received in step S107 as a search key to read out corresponding communication information (communication ID, relay device IP address) (step S108). Then, the transmitting / receiving unit 51 transmits response possible information to the starting terminal. The response possible information in this case includes the communication information read out in step S108. As a result, the transmitting / receiving unit 11 of the starting terminal receives the response possible information.

[0082] Next, the transmitting / receiving unit 91 of the destination terminal transmits establishment request information indicating a request to establish a mutual communication session to the IP address of the relay device included in the communication information received in step S105 (step S110). This establishment request information includes the communication ID of the starting terminal and the communication ID of the presenter of the destination terminal. The IP address of the destination terminal, which is the transmission source, is also transmitted. Meanwhile, the transmitting / receiving unit 11 of the starting terminal transmits establishment request information indicating a request to establish a mutual communication session to the IP address of the relay device included in the communication information received in step S109 (step S111). This establishment request information includes the communication ID of the starting terminal and the communication ID of the presenter of the destination terminal. The IP address of the starting terminal, which is the transmission source, is also transmitted. As a result, a communication session for a video call for mutual communication of images and sounds is established between the relay device 3 and the starting terminal (step S112-1), and a communication session for a video call for mutual communication of images and sounds is established between the relay device 3 and the destination terminal (step S112-2).

[0083] Next, the transmitting / receiving unit 11 of the initiating terminal transmits status information of its own terminal (initiating terminal) to the communication management device 5 (step S113-1). This status information includes the communication ID and operating status information (here, "in communication") of the initiating terminal. As a result, the transmitting / receiving unit 51 of the communication management device 5 receives the status information of the initiating terminal. Meanwhile, the transmitting / receiving unit 91 of the destination terminal transmits status information of its own terminal (destination terminal) to the communication management device 5 (step S113-2). This status information includes the communication ID and operating status information (here, "Chat (in communication)") of the destination terminal. As a result, the transmitting / receiving unit 51 of the communication management device 5 receives the status information of the destination terminal. In this way, each terminal notifies the communication management device 5 of its own operating status. Then, in the communication management device 5, the storage / read processing unit 59 changes the operating status associated with the communication ID of the initiating terminal to "Chat (communicating)" in the terminal status management DB (see FIG. 6(B)), and also changes the operating status associated with the communication ID of the destination terminal to "Chat (communicating)" (step S114). As a result, the initiating terminal (store terminal 1) and the destination terminal (presenter terminal 9) can make a video call by mutual communication, as shown in FIG. 2.

[0084] <Handling two-way input> Next, the process of two-way input between the store terminal 1 and the presenter terminal 9 will be described with reference to FIGS.

[0085] (Startup process for bidirectional input section) First, the startup process of the two-way input unit in the store terminal 1 and the presenter terminal 9 will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is a sequence diagram showing the process of starting up the two-way input unit. Fig. 13 is a diagram showing a display example of an input image on the presenter terminal side.

[0086] As shown in Fig. 12, when a presenter E wants a customer C at the store to input something such as a signature on an electronic document during a video call between the store terminal 1 and the presenter terminal 9, the presenter E presses the "Start two-way input" button 990 shown in Fig. 2, and the reception unit 92 receives the start of the two-way input unit 96 (S201). This starts the two-way input unit 96 (S202). Then, the display control unit 94 displays an input screen as shown in Fig. 13 on the display 906.

[0087] As shown in FIG. 13, a hideable task bar 920 is displayed at the top end of the input screen 900 displayed on the presenter terminal 9, and a hideable task bar 930 is displayed at the left end of the input screen 900.

[0088] Furthermore, the task bar 920 includes a file selection button 921 , a save button 922 , a previous page button 923 , thumbnails 924 of each page, a next page button 925 , an add page button 926 , a delete page button 927 , and an input disabled mode button 928 .

[0089] Among these, the file selection button 921 is a button for starting file selection so that the presenter can add (select) a file to be displayed on the input screen 900. This displays a file browser. The save button 922 is a button for saving the input stroke data and / or image data such as a document image in one file. The page back button 923 is a button for displaying the page one page before the page currently displayed on the input screen 900 on the input screen 900. The thumbnails 924 of each page are displayed, for example, as thumbnails of three pages. The page forward button 925 is a button for displaying the page one page ahead of the page currently displayed on the input screen 900 on the input screen. The page addition button 926 is a button for adding a new page in the initial state. The page deletion button 927 is a button for deleting the page currently displayed on the input screen 900. And the input disabled mode button 928 is a button for changing the input enabled mode, in which input to the input screen 100 displayed on the store terminal 1 side is possible, to the input disabled mode, in which input is not possible, for the store terminal 1 as the communication partner. The input enabled mode is an example of a first mode, and the input disabled mode is an example of a second mode.

[0090] On the other hand, the task bar 930 includes a pen button 931 , a highlighter button 932 , a marker button 933 , a partial eraser button 934 , a full eraser button 935 , an Undo button 936 , a Redo button 937 , a cursor button 938 , and a resize button 939 .

[0091] Among these, the pen button 931 is a button for enabling the presenter E to write (input) a stroke on the input screen 900 in the currently set color. The fluorescent button 932 is a button for enabling the presenter E to write (input) a stroke in a slightly thick fluorescent color in the currently set color. The marker button 933 is a button for enabling the presenter E to write (input) a marker that is displayed temporarily but automatically erased. The partial eraser button 934 is a button for enabling the presenter E to delete a stroke image that has been written once. In this case, only the part of the stroke image designated by the presenter E is deleted. The all eraser button 935 is a button for deleting all stroke images written (input) on the entire surface of the input screen 900 in one operation. The Undo button 936 is a button for returning the stroke image to the state before one operation. One stroke corresponds to one operation. The Redo button 937 is a button for returning the stroke image to the state one operation ahead. The cursor button 938 is a button for displaying a cursor on the input screen 900. This cursor can be used to point to a specific position on the input screen 900. The size change button 939 is a button for enlarging (or reducing) the display on the input screen 900 by pressing it to display candidates for enlargement and reduction ratios in a pull-down menu or the like, and by having the presenter E select a specific candidate.

[0092] Although the input screen 100 is also displayed on the store terminal 1 side, a task bar identical to the task bar 920 is not displayed on the input screen 100, and only a task bar 130 identical to the task bar 930 is displayed.

[0093] Next, the transmitting / receiving unit 91 transmits a request to start the two-way input unit 16 to the communication management device 5 (S204). This request includes a connection ID for identifying the mutual communication performed between the presenter terminal 9 and the store terminal 1 and the relay device 3. As a result, the communication management device 5 receives the request to start the two-way input unit 16.

[0094] Next, the transmitting / receiving unit 51 of the communication management device 5 transmits (or transfers) the start-up request for the bidirectional input unit 16 received in step S204 to the store terminal 1 (S205). As a result, the transmitting / receiving unit 11 of the store terminal 1 receives the start-up request for the bidirectional input unit 16 including the connection ID.

[0095] Next, in the store terminal 1, the two-way input unit 16 is started (S206). Then, the display control unit 14 displays the input screen 100 on the display 106 (S207).

[0096] Next, the transmitting / receiving unit 101 of the store terminal 1 transmits a startup completion notification to the communication management device 5, indicating that startup of the bidirectional input unit 16 has been completed (S208). This startup completion notification includes a connection ID for identifying communication between the store terminal 1 and the presenter terminal 9. As a result, the transmitting / receiving unit 51 of the communication management device 5 receives the startup completion notification.

[0097] Next, the transmitting / receiving unit 51 of the communication management device 5 transmits (transfers) a startup completion notification to the presenter terminal 9 (S209). As a result, the transmitting / receiving unit 91 of the presenter terminal 9 receives the startup completion notification. Then, in the presenter terminal 9, the display control unit 94 displays information that startup has been completed on the display 906 (S210).

[0098] (Sharing of document images) Next, a process for the presenter terminal 9 to share a material image with the store terminal 1 will be described with reference to Fig. 14 and Fig. 15. Fig. 14 is a sequence diagram showing the process for sharing a material image. In Fig. 15, (A) is a diagram showing a display image on the presenter terminal side, and (B) is a diagram showing a display image on the store terminal side.

[0099] As shown in Fig. 14, when the transmitting / receiving unit 91 of the presenter terminal 9 receives a designation of a material image such as a file name from the presenter E, the receiving unit 92 receives the designation of the material image (S221). As a result, the storage / reading processing unit 99 reads out the material image data from the storage unit 9000, and as shown in Fig. 15(A), the display control unit 94 displays the material image on the input screen 900 of the display 906 (S222). Here, the material image is an image of an individual membership application form.

[0100] Next, the transmitting / receiving unit 91 uploads the material image data to the communication management device 5 (S223). As a result, in the communication management device 5, the storage / read processing unit 59 stores the material image data in the storage unit 9000. Then, the transmitting / receiving unit 51 transmits an upload completion notification indicating that the upload is complete to the presenter terminal 9 (S225). As a result, the transmitting / receiving unit 91 of the presenter terminal 9 receives the upload completion notification.

[0101] Meanwhile, the transmitting / receiving unit 51 of the communication management device 5 transmits a download instruction to the store terminal 1 to cause the download of the material image (S226). This download instruction includes a URL (Uniform Resource Locator) indicating the storage location of the material image data. As a result, the transmitting / receiving unit 11 of the store terminal 1 receives the download instruction for the material image data.

[0102] Next, the store terminal 1 requests the URL received in step S226 to download the material image data (S227). As a result, in the communications management device 5, the storage / read processing unit 59 reads out the material image data from the storage area of ​​the storage unit 5000 indicated by the URL (S228). Then, the transmission / reception unit 51 downloads the material image data to the store terminal 1 (S229). As a result, the transmission / reception unit 11 of the store terminal 1 receives the material image data.

[0103] Next, in the store terminal 1, as shown in Fig. 15(B), the display control unit 14 causes the input screen 100 of the display 106 to display the same material image as that of the presenter terminal 9 (S230). Then, the transmitting / receiving unit 11 of the store terminal 1 transmits a display completion notification to the communication management device 5 indicating that the display of the input screen 100 has been completed (S231). As a result, the transmitting / receiving unit 51 of the communication management device 5 receives the display completion notification.

[0104] Next, in the communication management device 5, the transmitting / receiving unit 51 transmits (transfers) a display completion notification to the presenter terminal 9 (S232). As a result, the transmitting / receiving unit 91 of the presenter terminal 9 receives the display completion notification. Then, in the presenter terminal 9, the display control unit 94 displays information indicating that the display is completed on the input screen 900 of the display 906 (S233).

[0105] (Sharing of inputtable candidate areas) Next, a process in which the presenter terminal 9 shares an inputtable candidate area with the store terminal 1 will be described with reference to Figs. 15 to 18. Fig. 16 is a sequence diagram showing the inputtable candidate area sharing process. In Fig. 17, (A) is a diagram showing a display image on the presenter terminal side, and (B) is a diagram showing a display image on the store terminal side. In Fig. 18, (A) is a diagram showing a display image on the presenter terminal side, and (B) is a diagram showing a display image on the store terminal side.

[0106] 15(A), the presenter E presses the input-disabled mode button 928 with the electronic pen 90, and the reception unit 92 receives the change to the input-disabled mode (S241). As a result, the mode change unit 97 changes the mode from the input-enabled mode to the input-disabled mode (S242).

[0107] Next, as shown in FIG. 17(A), when the presenter E uses the electronic pen 90 to surround an area (inputable candidate area 940) that the presenter wants the customer C to input, the reception unit 92 receives the selection of the inputable candidate area 940 (S243). The inputable candidate area 940 is a candidate area into which input is possible even in the input-disabled mode, and at this point in time, input is not possible in the inputable candidate area 940, which is a candidate. As a result, the display control unit 94 displays the inputable candidate area 940 as shown in FIG. 17(A) (S244). Here, the outer edge of the inputable candidate area 940 is represented by a dashed line, but it may be a solid line, or the inside of the area may be displayed in a color different from the surrounding area. The same applies to the inputable candidate area 960 described later. Note that the data of the stroke coordinates is data indicating the outer edge of the inputable area (inputable candidate area). Furthermore, the display control unit 94 displays a lock icon 950, which visually indicates that the inputtable candidate area 940 is locked and cannot be input yet, near the inputtable candidate area 940 (S245). The lock icon 950 and an unlock icon 951, which will be described later, correspond to a specific table (for example, a table of stroke ID "S901") in the stroke information management DB 9001.

[0108] Next, in the presenter terminal 9, the calculation unit 98 calculates stroke coordinates of the inputtable candidate region 940 (S246). Then, the storage / read processing unit 99 stores the stroke information including the stroke coordinates as a new record in the stroke information management DB 9001 (see FIG. 5(C)) (S247). The stroke information stored in this step S247 includes a stroke data ID, a connection ID, an image ID, stroke coordinates, stroke color, stroke width, whether or not the region is closed, and whether or not input is possible, as shown in FIG. 5(C). Note that the storage / read processing unit 99 generates a stroke ID when storing the stroke coordinates and the like calculated in step S243 in the stroke information management DB 9001. Also, at this point, since the inputtable candidate region 940 is a closed region, whether or not it is closed is indicated as "1 (Yes)", and since it is locked, whether or not input is possible is indicated as "No".

[0109] Next, the transmitting / receiving unit 91 transmits a mode change instruction to the communications management device 5, which is an instruction to change from the input enabled mode to the input disabled mode (S248). This mode change instruction includes the stroke information stored in step S247. As a result, the transmitting / receiving unit 51 of the communications management device 5 receives the stroke information. Then, the storage / read processing unit 59 of the communications management device 5 stores the stroke information received in step S248 as a new record in the stroke information management DB 5006 (S249).

[0110] Next, the transmitting / receiving unit 51 of the communication management device 5 transmits (transfers) the mode change instruction received in step S248 to the store terminal 1 (S250). As a result, the transmitting / receiving unit 11 of the store terminal 1 receives the mode change instruction including the stroke information.

[0111] Next, in the store terminal 1, the storage / readout processing unit 19 stores the stroke information received in step S250 in the stroke information management DB 1001 as a new record (S251). Then, the mode change unit 17 changes the mode from the input possible mode to the input impossible mode based on the mode change instruction received in step S250 (S252). Then, as shown in FIG. 17(B), the display control unit 14 displays the input possible candidate area 140 corresponding to the input possible candidate area 940 on the input screen 100 of the display 106 at the same position as on the explainer terminal 9 side based on the stroke information (S253). In this case, the display control unit 14 displays the input possible candidate area 940 input on the input screen 900 as the input possible candidate area 140 in the same position on the input screen 100 with the same color and the same width based on the stroke information. Here again, the outer edge of the input possible candidate area 140 is represented by a broken line, but it may be a solid line, or the inside of the area may be displayed in a color different from the surrounding area. The same applies to an input possible area 160 described later. Furthermore, by the processes in steps S247, S249, and S251, the same stroke information can be shared among the three locations, that is, the store terminal 1, the communication management device 5, and the presenter terminal 9.

[0112] In FIG. 17(A), a lock icon corresponding to the lock icon 950 is not displayed on the store terminal 1 side, but it may be displayed.

[0113] As a result of the above, the enterable candidate area 940 entered and displayed on the presenter terminal 9 side is also displayed as the enterable candidate area 140 on the store terminal 1 side.

[0114] Furthermore, as shown in FIG. 18(A), when the presenter E inputs an inputtable candidate area 960 different from the inputtable candidate area 940, the processing of steps S241 to S251 described above is repeated, and the display control unit 94 displays the inputtable candidate area 960 and a lock icon 970, and the display control unit 14 displays the inputtable candidate area 160.

[0115] (Change to inputtable area) Next, the process of changing an inputtable candidate area to an inputtable area from the presenter terminal 9 to the store terminal 1 will be described with reference to Figs. 19 and 20. Note that an inputtable candidate area is an area in which input is not possible, and an inputtable area is an area in which input is possible. Fig. 19 is a sequence diagram showing the process of changing an inputtable candidate area to an inputtable area. In Fig. 20, (A) is a diagram showing a display image on the presenter terminal side, and (B) is a diagram showing a display image on the store terminal side.

[0116] First, when the presenter E presses the lock icon 950 shown in Fig. 18(A) with the electronic pen 90, the reception unit 92 receives input permission for the inputtable candidate area 940 (S261). As a result, the display control unit 94 changes the display from the lock icon 950 to an unlock icon 951 as shown in Fig. 20(A) and changes the display from the inputtable candidate area 940 to the inputtable area 941 as shown in Fig. 20(A), thereby displaying the input permission (S262). Here, the outer edge of the inputtable area 941 is represented by a solid line, but it may be a dashed line, or the inside of the area may be displayed in a color different from the surrounding area. However, if the lock icons 950, 970 and the unlock icon 951 are not displayed, the inputtable candidate areas 940, 960 cannot be distinguished from the inputtable area 941, so it is desirable to distinguish the display forms by, for example, showing the inputtable candidate area 960 with a dashed line and the inputtable area 941 with a solid line, as shown in Fig. 20(A). In this way, it is desirable that the inputtable candidate area 940 and the inputtable area 941 differ in at least one of the shape and color. Note that the inputtable candidate area 940 is an example of a first display form, and the inputtable area 941 is an example of a second display form.

[0117] Next, the storage / read processing unit 99 changes the "input possible" column of the record corresponding to the changed display of the unlock icon 951 in the stroke information management DB 9001 from "not possible" to "possible" (S263).

[0118] Next, the transmitting / receiving unit 91 of the presenter terminal 9 transmits change information indicating the change made in step S263 ("OK") to the communication management device 5 (S264). This change information includes the connection ID of the record changed in step S263. As a result, the transmitting / receiving unit 51 of the communication management device 5 receives the change information.

[0119] Next, in the communication management device 5, similar to the process of step S263, the storage / read processing unit 59 changes the "Input Possible" field of the record corresponding to the changed unlock icon 951 (i.e., the connection ID received in step S264) in the stroke information management DB 5006 from "No" to "Yes" (S265). Then, the transmitting / receiving unit 51 transmits (transfers) the change information received in step S264 to the store terminal 1 (S266). As a result, the transmitting / receiving unit 11 of the store terminal 1 receives the change information.

[0120] Next, in the store terminal 1, similarly to the processes of steps S263 and S265, the storage / readout processing unit 19 changes the "input permission" column of the record corresponding to the unlock icon 951 (i.e., the connection ID received in step S266) whose display has been changed in the stroke information management DB 1001 from "not permitted" to "permitted" (S267). Then, the reception unit 12 enables the reception of input by the electronic pen 10 to the inputtable candidate area 140 as the specific area (S268). In this case, the reception unit 12 plays the role of an input permission unit. Also, the display control unit 14 changes the display from the inputtable candidate area 140 shown in FIG. 18(B) to the inputtable area 141 shown in FIG. 20(B), thereby displaying the input permission (S268). At this time, the presenter E may inform the presenter that the inputtable area 141 is now in a state where input is permitted by video call, just to be sure. Here too, the outer edge of the inputtable area 141 is represented by a solid line, but it may be a dashed line, or the inside of the area may be displayed in a color different from the surroundings. However, since the inputtable candidate areas 140, 160 and the inputtable area 141 cannot be distinguished from each other, it is preferable to distinguish the display form by representing the inputtable area 141 by a solid line and the inputtable candidate area 160 by a dashed line, as shown in FIG. 20(B). In this way, it is preferable that the inputtable area 141 and the inputtable candidate area 160 differ in at least one of the shape and color. Note that the inputtable candidate area 140 is an example of a first display form, and the inputtable area 141 is an example of a second display form.

[0121] From the above, customer C can understand that it is now possible to input his / her name into the input area 141 with the electronic pen 10.

[0122] (Sharing of input contents) Next, a process of sharing input contents from the store terminal 1 to the presenter terminal 9 will be described with reference to Fig. 21 and Fig. 22. Fig. 21 is a sequence diagram showing the process of sharing input contents. In Fig. 22, (A) is a diagram showing a display image on the presenter terminal side, and (B) is a diagram showing a display image on the store terminal side.

[0123] First, as shown in FIG. 22(B), when the customer C inputs desired information (here, the name of the customer C) into the inputtable area 141 using the electronic pen 10, the reception unit 12 receives the input contents (S281). As a result, the display control unit 14 displays the input contents in the inputtable area 141 as shown in FIG. 22(B) (S282). Then, the calculation unit 18 calculates the stroke coordinates of the stroke input into the inputtable area 141 (S283). Then, the storage / read processing unit 19 stores the stroke information including the stroke coordinates as a new record in the stroke information management DB 1001 (see FIG. 5(C)) (S284). The stroke information stored in this step S284 includes the stroke data ID, connection ID, image ID, stroke coordinates, stroke color, stroke width, and whether or not the area is closed, as shown in FIG. 5(C). In this case, the storage / read processing unit 19 also generates a new stroke ID. When input is made on the store terminal 1 side, the stroke information does not include whether input is possible or not.

[0124] Next, the transmission / reception unit 11 transmits the stroke information stored in step S284 to the communication management device 5 (S285). As a result, the transmission / reception unit 51 of the communication management device 5 receives the stroke information. Then, the storage / read processing unit 59 of the communication management device 5 stores the stroke information received in step S285 as a new record in the stroke information management DB 5006 (S286).

[0125] Next, the transmitting / receiving unit 51 of the communication management device 5 transmits (transfers) the stroke information received in step S285 to the presenter terminal 9 (S287). As a result, the transmitting / receiving unit 91 of the presenter terminal 9 receives the stroke information.

[0126] Next, in the presenter terminal 9, the storage / readout processing unit 99 stores the stroke information received in step S287 in the stroke information management DB 9001 as a new record (288). Then, as shown in FIG. 17(B), the display control unit 14 displays input information corresponding to the input contents (here, name) on the store terminal 1 side on the input screen 900 of the display 906 at the same position as on the store terminal 1 side on the input screen 900 based on the stroke information received in step S250 (S289). In this case, the display control unit 94 displays the input contents input on the input screen 100 in the same position on the input screen 900 with the same color and width according to the stroke information. Also, by the processes of the above steps S284, S286, and S287, the same stroke information can be shared at three locations, the store terminal 1, the communication management device 5, and the presenter terminal 9.

[0127] As a result, the input contents input and displayed on the store terminal 1 side are displayed on the presenter terminal 9 side in the same format.

[0128] [Main Effects of the Present Embodiment] As described above, according to this embodiment, the input mode allows input to the entire displayed predetermined area, and the input mode prevents input to the entire predetermined area, thereby preventing the customer C from inputting data without permission. Furthermore, in the input-disabled mode, input to a specific area of ​​the predetermined area is enabled, so that the customer C can grasp the specific area desired by the presenter E even in the case of a remote video call. This has the effect of preventing the discussion between the presenter E and the customer C from taking too long. In this case, the "entire predetermined area" may be the entire input screen 100, or the entire document image displayed on a part of the input screen 100.

[0129] 〔supplement〕 The store terminal 1 is an example of a communication terminal. The store terminal 1 includes not only a PC but also a smart watch, a game machine, a device dedicated to video calling, and the like. The presenter terminal is an example of another communication terminal.

[0130] Each of the components such as the CPUs 101, 301, 501, 901, etc. may be a single component or a plurality of components.

[0131] Each function in the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU, and a conventional circuit module designed to execute each of the above-described functions. [Explanation of symbols]

[0132] 1. Store terminal (an example of a communication terminal) 3. Repeater 5 Communication management device 9. Presenter terminal (an example of another communication terminal) 11 Transmitting / receiving unit (an example of a transmitting means, an example of a receiving means) 12 Reception unit (an example of an input enabling means) 13 Communication control section 14 Display control unit (an example of a display control means) 15 Judgment Department 16 Bidirectional input section 17 Mode change unit (an example of a mode change means) 18 Calculation section 100 Communication Network [Prior art documents] [Patent documents]

[0133] [Patent Document 1] JP 2008-154059 A

Claims

1. A communication terminal that shares input information with other communication terminals, a mode change means for changing a mode from a first mode in which input to the entirety of a displayed predetermined area is possible to a second mode in which input to the entirety of the predetermined area is impossible; A receiving means for receiving a mode change instruction transmitted from the other communication terminal; an input enabling means for enabling an input to a specific area of ​​the predetermined area in the second mode; having the instruction to change the mode includes information on an inputtable candidate area that is a candidate for the specific area, the mode change means changes the mode from the first mode to the second mode based on the instruction to change the mode received by the receiving means, The receiving means receives change information transmitted by the other communication terminal pressing an icon corresponding to the inputtable candidate area, The communication terminal is characterized in that the input enabling means enables input into an input possible candidate area corresponding to the icon pressed by the other communication terminal based on the change information received by the receiving means.

2. 2. The communication terminal according to claim 1, further comprising a display control means for displaying an outer edge of the inputtable candidate area.

3. 3. The communication terminal according to claim 2, characterized in that after the display control means displays the outer edge of the inputtable candidate area in a first display form when input into the inputtable candidate area is not possible, the display control means changes the outer edge from the first display form to a second display form when the input enabling means enables input into the inputtable candidate area.

4. 4. The communication terminal according to claim 3, wherein the first display mode and the second display mode are different in at least one of shape and color.

5. The communication terminal described in Claim 2, characterized in that the display control means displays the icon that has not been pressed by the other communication terminal and the icon that has been pressed by the other communication terminal in different display forms.

6. 2. The communication terminal according to claim 1, wherein the other communication terminal transmits an instruction for the mode change by accepting the mode change.

7. The communication terminal described in Claim 2, characterized in that the display control means displays the outer edge of the inputtable candidate area based on data indicating the outer edge of the inputtable candidate area contained in the information of the inputtable candidate area.

8. A communication terminal according to claim 1; The other communication terminal; A communication system comprising:

9. A sharing method executed by a communication terminal for sharing input information with another communication terminal, comprising: a mode change step of changing a mode from a first mode in which an input to the entirety of a displayed predetermined area is possible to a second mode in which an input to the entirety of the predetermined area is not possible; a receiving step of receiving a mode change instruction transmitted by the other communication terminal; an input enabling step of enabling an input to a specific area of ​​the predetermined area in the second mode; having the instruction to change the mode includes information on an inputtable candidate area that is a candidate for the specific area, the mode changing step changes the mode from the first mode to the second mode based on the instruction to change the mode received by the receiving step, The receiving step receives change information transmitted by pressing an icon corresponding to the inputtable candidate area from the other communication terminal, A sharing method characterized in that the input enabling step enables input into an input possible candidate area corresponding to the icon pressed by the other communication terminal based on the change information received by the receiving step.

10. A program for causing a computer to execute the method according to claim 9.

Citation Information

Patent Citations

  • Control device, method and program for controlling operator base

    JP2008154059A

  • Equipment management system and equipment management program

    JP2015022492A

  • Information processing apparatus and program

    JP2018005273A