Communication terminal, information and communication system, image processing method and program

The communication terminal automatically captures the screen based on input duration or distance to maintain image-screen relationships, addressing the issue of inconsistent image processing due to user forgetfulness.

JP7581683B2Active Publication Date: 2024-11-13RICOH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2020124351
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-07-21
Publication Date
2024-11-13
Estimated Expiration
2040-07-21

AI Technical Summary

Technical Problem

Communication terminals struggle to maintain the relationship between user-input images and the displayed screen content when the screen changes due to the user forgetting to operate capture selection means, leading to inconsistencies in image processing.

Method used

A communication terminal that automatically captures the screen when predetermined conditions are met, such as a specific duration of input or distance, ensuring the relationship between user-input images and the screen content is maintained even if the screen changes.

Benefits of technology

Ensures consistent display of user-input images by automatically capturing the screen when necessary, preventing inconsistencies caused by user oversight in selecting capture options.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581683000001
    Figure 0007581683000001
  • Figure 0007581683000002
    Figure 0007581683000002
  • Figure 0007581683000003
    Figure 0007581683000003
Patent Text Reader

Abstract

To solve a problem of a conventional electronic blackboard that, if a user forgets to press a camera button ba1 for capturing the screen of a PC2 as an image and continues to input stroke images, the relationship between the display content of the screen displayed on the electronic blackboard 1 and the content of the stroke images cannot be maintained when the screen of PC2 changes.SOLUTION: A determination unit 15a of an electronic blackboard 1a determines whether the length or time of a pen drawing line is greater than or equal to a threshold (step S101-21). When it is greater than the predetermined threshold, an image processing unit 17a captures a PC screen (step S101-3), and then creates, from the images constituting the captured PC screen, display data for the captured screen to write to these images (step S101-4). After that, a display control unit 14a displays the display data for the captured screen created in step S101-4 on a display 180 (step S101-5).SELECTED DRAWING: Figure 23
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a communication terminal, an information communication system, a communication method, and a program. [Background technology]

[0002] Communication systems that communicate with remote locations via a communication network such as the Internet are becoming widespread. Examples of such communication systems include conference systems that hold remote conferences in companies, educational institutions, government agencies, and the like. In addition, the conference system uses communication terminals such as electronic whiteboards that display an image on a display and allow a user to draw stroke images such as letters, numbers, and figures on the image (see, for example, Patent Document 1). The stroke image is drawn by the communication terminal electronically converting the content drawn by the user by touching and moving the display with an electronic pen or his / her hand, and generating stroke data such as coordinate data. Furthermore, when a stroke image is drawn on one communication terminal, the stroke data for reproducing the stroke image can be transmitted to the other communication terminal via the communication network, so that the same stroke image can be displayed on the other communication terminal.

[0003] In addition to the above-mentioned technology, when capturing a screen displaying an image such as a stroke image, a technology is known in which, when the entire capture button 422 is pressed out of the entire capture button 422 and partial capture button 423 included in a pop-up menu 421, the entire screen 41 of the document display application 11 is captured (for example, see Patent Document 2). Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional communication terminal, the image transmitted from the information processing device communicating with the communication terminal is not is displayed. Display the screen, theWhen capturing a screen, the user must operate the capture selection means, such as the camera button, set on the communication terminal in advance. However, if the user forgets to operate the capture selection means, On the screen Input image continues to be input. Be In this case, when the image transmitted by the information processing device changes, the content on the screen of the communication terminal also changes, resulting in a problem that the relationship between the content of the input image input by the user and the content of the displayed screen cannot be maintained. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention relates to a method for detecting image data transmitted by an information processing device. Represented by image is displayed. A communication terminal that displays a screen, The above screen Draw by inputting the coordinates above a receiving means for receiving an input image that has been input by the input means, and if the input of the input image that has been received by the receiving means satisfies a predetermined condition, an image represented by the image data; The input image and Including The above and a capture means for capturing a screen. Effect of the Invention

[0006] As described above, according to the present invention, even if the content of the displayed screen changes, it is possible to maintain the relationship with the content of the input image input onto the screen. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a schematic diagram showing an example of a communication route according to the embodiment; [Diagram 2] FIG. 13 is a diagram showing an example of an image of how an electronic whiteboard is used; [Diagram 3]FIG. 1A is a diagram showing an example of a layer configuration of each image displaying a PC screen on an electronic whiteboard, and FIG. 1B is a diagram showing an example of a layer configuration of each image displaying an imported screen on an electronic whiteboard. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of the electronic whiteboard according to the present embodiment. [Diagram 5] 2 is a diagram illustrating an example of the hardware configuration of a PC, a relay device, a communication management system, and an image storage device according to the present embodiment. [Figure 6] 1 is a diagram illustrating an example of an overall configuration of a communication system according to an embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating an example of each functional block of the communication system according to the present embodiment. [Figure 8] FIG. 13 is a conceptual diagram illustrating an example of an authentication management table. [Figure 9] FIG. 13 is a conceptual diagram illustrating an example of a terminal management table. [Figure 10] FIG. 13 is a conceptual diagram illustrating an example of a destination list management table. [Figure 11] FIG. 13 is a conceptual diagram illustrating an example of a session management table. [Figure 12] FIG. 13 is a conceptual diagram illustrating an example of a relay device management table. [Figure 13] FIG. 11 is a sequence diagram showing an example of a preparatory process for starting remote communication between electronic whiteboards. [Figure 14] FIG. 13 is a diagram showing an example of a destination list screen displayed on the electronic whiteboard 1a. [Figure 15] FIG. 11 is a sequence diagram illustrating an example of a process for starting remote communication. [Figure 16] 10 is a sequence diagram showing an example of a process between the PC 2 and the electronic whiteboard 1. FIG. [Figure 17] 11 is a sequence diagram illustrating an example of a process for sharing image data of a document image and stroke data of a stroke image. FIG. [Figure 18] 11 is a sequence diagram showing an example of uploading and downloading processes of image data of a document image. FIG. [Figure 19]1A is a diagram showing an example of a screen of the electronic whiteboard 1a, FIG. 1B is an example of a screen of the electronic whiteboard 1b, FIG. 1C is an example of a screen of the electronic whiteboard 1a, and FIG. 1D is an example of a screen of the electronic whiteboard 1b. [Figure 20] 10 is a flowchart illustrating an example of a display switching process according to the embodiment. [Figure 21] (A) is an example of a PC screen on the electronic whiteboard 1a, (B) is an example of a pen mode screen on the electronic whiteboard 1a, (C) is an example of a screen on which a stroke image is drawn on the PC screen of the electronic whiteboard 1a, and (D) is an example of a marker mode screen on the electronic whiteboard 1a. [Figure 22] 6 is a flowchart showing an example of a display switching process and a mode switching process according to the embodiment. [Figure 23] 10 is a flowchart illustrating an example of a determination process at the time of capturing a screen according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that if there is a duplicated portion in the description of the drawings, the description will be omitted.

[0009] [Outline of the communication system] <Communication route> First, a communication system for conducting a video conference while drawing between multiple electronic whiteboards 1a and 1b will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing an example of a communication route according to this embodiment. Note that this may be called a "television conference" rather than a "video conference." Also, a video conference will be described here as an example, but it may also be a meeting, simple conversation, etc.

[0010] The communication system is constructed by multiple electronic whiteboards 1a, 1b, a relay device 3, a communication management system 5, and an image storage device 7. The electronic whiteboards 1a, 1b communicate with each other content data such as video data and audio data for calls, and image data and stroke data for sharing. The stroke data is data required to play (reproduce) a stroke image, and includes coordinate data, line width data, line color data, vector data, etc. The electronic whiteboards 1a, 1b transmit and receive video data and audio data for calls, and play the video and audio of the other party, enabling remote video calls.

[0011] The electronic whiteboards 1a and 1b transmit and receive image data of a shared material image, allowing participants using the communication system to share the same material image. The material image is an image displayed on the display of the electronic whiteboards 1a and 1b, and may be an image such as a conference material, a background image displayed on the display, or a captured screen when the display screen is captured. In this embodiment, the "material image" may include video as a moving image, but for ease of explanation, it will be simply referred to as a "material image." Therefore, the data showing the material image includes material image data and material video data.

[0012] Moreover, the electronic whiteboards 1a and 1b transmit and receive stroke data of a shared stroke image, allowing participants using the communication system to share the same stroke image. A stroke image is an image drawn by a user using an electronic pen or the like. Specifically, a stroke image is an image represented by points or lines drawn by handwritten strokes. Moreover, a stroke image is displayed by stroke data indicating points that specify coordinates on the display of the electronic whiteboards 1a and 1b.

[0013] The communication system is not limited to the two electronic whiteboards 1a and 1b, but may be constructed using three or more electronic whiteboards. Hereinafter, the electronic whiteboards 1a and 1b will be collectively referred to as "electronic whiteboard 1."

[0014] 1, an electronic whiteboard equipped with a video conference function is shown as an example of the electronic whiteboards 1a and 1b. Note that the image of the video data may be either a moving image or a still image.

[0015] Moreover, an electronic whiteboard that is the request source that requests the start of a video conference is represented as an "initiating terminal," and an electronic whiteboard that is the destination (relay destination) of the request is represented as a "destination terminal." In FIG. 1, electronic whiteboard 1a is represented as the initiating terminal, and electronic whiteboard 1b is represented as the destination terminal. However, when a request to start a video conference is made from electronic whiteboard 1b, electronic whiteboard 1b becomes the initiating terminal, and electronic whiteboard 1a becomes the destination terminal. Each of electronic whiteboards 1a and 1b may be used in any environment, including communication between multiple business locations, communication between different rooms in the same business location, communication within the same room, communication between indoors and outdoors, and communication between outdoors and outdoors.

[0016] PC (Personal Computer) 2 is a computer connected to each of the electronic whiteboards 1a and 1b, and transmits video (image) data, audio data, etc., which become material videos (images) to the electronic whiteboards 1a and 1b. PC 2 and the electronic whiteboards 1a and 1b are connected to each other by an interface cable such as VGA (registered trademark, omitted below) or HDMI (registered trademark, omitted below). PC 2 functions as an example of an information processing device in this embodiment.

[0017] The relay device 3 is configured by a computer, and performs a process of relaying content data for communication or phone calls between the interactive whiteboards 1a and 1b.

[0018] The communication management system 5 is configured by a computer and centrally manages login authentication from the electronic whiteboards 1a and 1b, management of the communication status of the electronic whiteboards 1a and 1b, management of the destination list, and the communication status of the relay device 3. In addition, the communication management system 5 relays shared stroke data between the multiple electronic whiteboards 1a and 1b.

[0019] The image storage device 7 is configured by a computer, and stores image data of a shared document image uploaded from the electronic whiteboard 1a, and downloads the image data to the electronic whiteboard 1b. The reverse is also possible. That is, the image storage device 7 can store image data of a shared document image uploaded from the electronic whiteboard 1b, and download the image data to the electronic whiteboard 1a.

[0020] In addition, the relay device 3, the communication management system 5, and the image storage device 7 may be constructed by a single computer, or may be constructed by multiple computers in which each part (function or means) is divided and assigned arbitrarily.

[0021] In addition, in the communication system, a management information session sei is established between the electronic whiteboards 1a and 1b via the communication management system 5 for transmitting and receiving various management information. Furthermore, four sessions are established between the electronic whiteboards 1a and 1b via the relay device 3 for transmitting and receiving four types of data, high-resolution image data, medium-resolution image data, low-resolution image data, and sound data. In FIG. 1, these four sessions are collectively shown as an image and sound data session sed. Note that the image and sound data session sed does not necessarily have to be four sessions, and may be fewer or more than four sessions. Furthermore, a communication session may be established directly between the initiating terminal and the destination terminal without the relay device 3.

[0022] Furthermore, in the communication system, stroke data can be transmitted and received between the electronic whiteboards 1a and 1b by using the management information session sei.

[0023] Here, the resolution of the video data handled in this embodiment will be described. The low-resolution video data is, for example, 120 pixels long and 160 pixels wide, and serves as a base image. The medium-resolution video data is, for example, 240 pixels long and 320 pixels wide. The high-resolution video data is, for example, 480 pixels long and 640 pixels wide. Among these, when passing through a narrowband path, low-quality video data consisting only of low-resolution video data serving as a base image is relayed. When the band is relatively wide, low-quality video data consisting of low-resolution video data serving as a base image and medium-quality video data consisting of medium-resolution video data is relayed. When the band is very wide, high-quality video data consisting of low-resolution video data serving as a base image, medium-resolution video data, and high-resolution video data is relayed. Note that the amount of sound data is smaller than that of video data, and therefore sound data can be relayed even through a narrowband path.

[0024] <Image of using the electronic whiteboard> Fig. 2 is a diagram showing an example of an image of how the electronic whiteboard is used. As shown in Fig. 2, the electronic whiteboard 1 is composed of a leg 151 having a plurality of casters on its lower side, a support 152 provided on the upper side of the leg 151, a main body 153 of the electronic whiteboard 1 provided on the upper side of the support 152, and a display 180 provided on the front side of the main body 153. The main body 153 has a built-in CPU 101, which will be described later, and the like. A user can input (draw) a stroke image such as a character on the display 180 using an electronic pen 190 or the like.

[0025] <Layer configuration of images displayed on the electronic whiteboard> FIG. 3 is a diagram showing a layer configuration of an image displayed on an electronic whiteboard. FIG. 3(A) is a diagram showing an example of a layer configuration of each image displaying a PC screen on an electronic whiteboard, and FIG. 3(B) is a diagram showing an example of a layer configuration of each image displaying an imported screen on an electronic whiteboard. Here, the PC screen is a screen when an image or video transmitted from a PC2 connected to the electronic whiteboard 1 is displayed on the display 180 of the electronic whiteboard 1, and is also called a first screen. On the other hand, the imported screen is a captured screen when a display screen of the PC2 is subjected to a screen import operation (screen capture operation) by a user, and is also called a second screen. For convenience of explanation, in this embodiment, the image or video transmitted from the PC2 is simply called an "image".

[0026] As shown in FIG. 3(A), the electronic whiteboard 1 creates an image by superimposing (composing) four image layers. Each image layer is composed of, from the left, a UI image (L1), a shared stroke image (L2), a document image (L3), and a background image (L4) in the order that the user can see even when overlapped. Of these, the UI image (L1) layer and the background image (L4) layer are image layers that are displayed only on the electronic whiteboard 1a, while the shared stroke image (L2) layer and the document image (L3) layer are image layers that are displayed and shared on the other party's electronic whiteboard 1b. In addition to this image layer, a picture of the other party's base is displayed by picture-in-picture technology. Each image layer is explained below. Note that the word "layer" is a katakana spelling of "Layer" and refers to a structure or state in which images can be used by overlapping them like cel drawings. Therefore, in Japanese, the word "layer" means a layer or a superposition.

[0027] The layer of the UI image (L1) is an image layer that displays buttons, icons, etc. and accepts UI (User Interface) operations such as input or selection so that the user can make a predetermined input or selection with the electronic pen 190 or the hand H. Here, as an example, a "shared" icon a11 that is pressed or otherwise operated (hereinafter also simply referred to as "operation") to select the shared mode, and a "non-shared" icon a12 that is operated to select the non-shared mode are displayed. In addition, the UI screen (L1) also displays buttons, icons, etc. that set the color, width, etc. of the line drawn by the electronic pen 190.

[0028] The layer of the shared stroke image (L2) is an image layer when a user shares a stroke image drawn by the electronic pen 190 or hand H with the other electronic whiteboard 1b (in the case of "shared mode"). In this case, the electronic whiteboard 1a can display the same stroke image not only on the electronic whiteboard 1a, which is the own terminal, but also on the electronic whiteboard 1b, which is the other terminal, by sending stroke data to the other electronic whiteboard 1b via the communication network 100. Here, as an example, a stroke image b showing a curved arrow is displayed.

[0029] The material image (L3) layer is an image layer for displaying a material image that is an output image from the PC 2 or the like connected to the electronic whiteboard 1a. Here, a material image c1 showing a pie chart is displayed.

[0030] The background image (L4) layer is an image layer for displaying a background image specific to each electronic whiteboard 1. Unlike stroke images and document images, background images are images that are not changed fundamentally, and are images used to improve the overall appearance of the image displayed on the display 180. Here, as an example, a background image d1 such as a grid image including the characters "Ri-Photo Electronic Whiteboard" that are always displayed on the display 180 is displayed.

[0031] Similarly, four image layers are displayed on the electronic whiteboard 1b. In this case, the UI image (L1) and the background image (L4) are image layers that are displayed only on the electronic whiteboard 1b, and the shared stroke image (L2) and the document image (L3) are image layers that are also displayed on the other electronic whiteboard 1a and shared.

[0032] On the other hand, in Fig. 3(B), the electronic whiteboard 1 also creates an image in which four image layers are superimposed (composite). The image layers are configured as follows, from the left, in the order in which the user can see them even when they overlap: UI image (L1), shared stroke image (L2), document image (L3), and background image (L4). However, Fig. 3(B), which displays the import screen of the electronic whiteboard, differs from Fig. 3(A) in that the document image c1 is hidden (as shown by the dotted line) in the document image (L3) layer, and the document image c1 is displayed copied onto the background image d1. The screen generation of the PC screen and the import screen described above will be explained in detail later.

[0033] [Hardware configuration] Next, the hardware configuration according to this embodiment will be described.

[0034] <Hardware configuration of the electronic whiteboard> Fig. 4 is a diagram showing an example of a hardware configuration of the electronic whiteboard according to the present embodiment. As shown in Fig. 4, the electronic whiteboard 1 includes a CPU (Central Processing Unit) 101, a ROM (Read only Memory) 102, a RAM (Random Access Memory) 103, an SSD (Solid State Drive) 104, a network I / F 105, and an external device connection I / F (Interface) 106.

[0035] Of these, the CPU 101 controls the overall operation of the electronic whiteboard 1. The ROM 102 stores the CPU 101 and programs used to drive the CPU 101, such as an IPL (Initial Program Loader). The RAM 103 is used as a work area for the CPU 101. The SSD 104 stores various data, such as programs for the electronic whiteboard. The network controller 105 controls communication with the communication network 100. The external device connection I / F 106 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 130 and external devices (a microphone 140, a speaker 150, and a camera 160).

[0036] The electronic whiteboard 1 also includes a capture device 111, a GPU 112, a display controller 113, a contact sensor 114, a sensor controller 115, an electronic pen controller 116, a short-range communication circuit 119, an antenna 119a of the short-range communication circuit 119, a power switch 122, and selection switches 123.

[0037] Of these, the capture device 111 is hardware that displays video information as a still image or a moving image on the display of the PC 2 connected to the electronic whiteboard 1. The capture device 111 is also called a capture board, and is connected to the PC 2 via, for example, a USB I / F. The GPU (Graphics Processing Unit) 112 is a semiconductor chip that specializes in graphics. The display controller 113 controls and manages the screen display in order to output an output image from the GPU 112 to the display 180 or the like. The contact sensor 114 detects that the electronic pen 190, the user's hand H, or the like has come into contact with the display 180. The sensor controller 115 controls the processing of the contact sensor 114. The contact sensor 114 inputs and detects coordinates using an infrared blocking method. The input method and detection method of the coordinates are such that two light receiving / emitting devices installed at both ends of the upper side of the display 180 emit a plurality of infrared rays parallel to the display 180, and receive the light that is reflected by a reflecting member provided around the display 180 and returns along the same optical path as the optical path of the light emitted by the light receiving element. The contact sensor 114 outputs the ID of the infrared rays emitted by the two light receiving / emitting devices that are blocked by an object to the sensor controller 115, and the sensor controller 115 specifies the coordinate position that is the contact position of the object. The electronic pen controller 116 determines whether the pen tip or the pen tail is in contact with the display 180 by communicating with the electronic pen 190. The short-distance communication circuit 119 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 122 is a switch for switching the power of the electronic whiteboard 1 on / off. The selection switches 123 are, for example, a group of switches for adjusting the brightness and color of the display 180.

[0038] Furthermore, the electronic whiteboard 1 includes a bus line 110. The bus line 110 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 101 shown in FIG.

[0039] Note that the contact sensor 114 is not limited to the infrared interruption method, and various detection means such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive touch panel that identifies the contact position by a voltage change between two opposed resistive films, and an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction generated when a contact object touches the display unit may be used. Further, the electronic pen controller 116 may be configured to determine the presence or absence of contact not only at the tip and the butt of the electronic pen 190 but also at the portion held by the user of the electronic pen 190 and other portions of the electronic pen 190. As described above, in the present embodiment, the electronic blackboard 1 functions as an example of a communication terminal.

[0040] <Hardware configurations of a PC, a relay device, a communication management system, and an image storage device> FIG. 5 is a diagram showing an example of the hardware configurations of a PC, a relay device, a communication management system, and an image storage device according to the present embodiment. A computer as an example of the PC 2 includes a CPU 201, a ROM 202, a RAM 203, an HD (Hard Disk) 204, an HDD (Hard Disk Drive) controller 205, a display 206, an external device connection I / F (Interface) 208, a network I / F 209, a keyboard 211, a pointing device 212, a DVD-RW (Digital Versatile Disk ReWritable) drive 214, a media I / F 216, and a bus line 210.

[0041] Among these, the CPU 201 controls the operation of the entire PC 2. The ROM 202 stores programs such as an IPL used to drive the CPU 201. The RAM 203 is used as a work area for the CPU 201. The HD 204 stores various data such as information processing programs. The HDD controller 205 controls the reading or writing of various data from the HD 204 under the control of the CPU 201. The display 206 displays various information such as a cursor, a menu, a window, characters, or an image. The external device connection I / F 208 is an interface for connecting various external devices. The network I / F 209 is an interface for data communication using a communication network such as the Internet. The keyboard 211 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, and the like. The pointing device 212 is a type of input means such as a mouse for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 214 controls the reading or writing (storing) of data from the DVD-RW. The DVD-RW drive 214 may control reading or writing (storing) of data from or to a disc such as a BD-RE (Blu-ray Disc Rewritable) instead of a DVD-RW drive. The media I / F 216 controls reading or writing (storing) of data from or to a recording medium 215 such as a flash memory.

[0042] The bus line 210 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 201 shown in FIG.

[0043] 1 are similar to the hardware configuration of PC 2, so their explanation will be omitted. However, in the case of relay device 3, HD 304 stores a relay program, etc., in the case of communication management system 5, HD 504 stores a communication management program, etc., in the case of image storage device 7, HD 704 stores an image storage program, etc.

[0044] [Overall configuration of communication system] Next, the overall configuration of the communication system will be described with reference to Fig. 6. Fig. 6 is a diagram showing the overall configuration of the communication system according to the embodiment of the present invention.

[0045] In Fig. 6, electronic whiteboard 1a is installed at site A, and electronic whiteboard 1b is installed at site B. For example, site A is the Tokyo office in Japan, and site B is the Beijing office in China. At site A, user A1 uses electronic whiteboard 1a, and at site B, users B1 and B2 use electronic whiteboard 1b.

[0046] Furthermore, the electronic whiteboards 1a and 1b, the PCs 2a and 2b, the relay device 3, the communication management system 5, and the image storage device 7 can communicate data with each other via a communication network 100 such as the Internet. The communication network 100 may include a wireless communication portion.

[0047] In Fig. 1 and Fig. 6, the electronic whiteboard 1a is an electronic whiteboard capable of video communication with the PC 2a. A system including the electronic whiteboard 1a and the PC 2a is referred to as an information communication system 8a. In Fig. 1 and Fig. 6 described above, the PC 2a is installed outside the site A, but may be used in the same site A as the electronic whiteboard 1a. The other electronic whiteboard 1b is an electronic whiteboard capable of video communication with the PC 2b. A system including the electronic whiteboard 1b and the PC 2b is referred to as an information communication system 8b. As with the above-mentioned PC 2a, in Fig. 1 and Fig. 6, the PC 2b is installed outside the site B, but may be used in the same site B as the electronic whiteboard 1b. The information communication system 8a and the information communication system 8b are collectively referred to simply as the information communication system 8.

[0048] [Functional configuration of communication system] Next, the functional configuration of this embodiment will be described with reference to Fig. 7 to Fig. 12. Fig. 7 is a diagram showing an example of each functional block of the communication system according to this embodiment.

[0049] <Functional configuration of the electronic whiteboard 1a> As shown in Fig. 7, the electronic whiteboard 1a has a transmitting / receiving unit 11a, a receiving unit 12a, a video / audio processing unit 13a, a display control unit 14a, a determining unit 15a, a measuring unit 16a, an image processing unit 17a, a providing / obtaining unit 18a, and a storing / reading processing unit 19a. Each of these units is a function or a functioning means realized by any of the components shown in Fig. 4 operating in response to an instruction from the CPU 101 according to a program loaded from the SSD 104 onto the RAM 103. The electronic whiteboard 1a also has a storage unit 1000a constructed by the RAM 103 and the SSD 104 shown in Fig. 4.

[0050] <<Functional configuration of the electronic whiteboard 1a>> Next, each component of the electronic whiteboard 1a will be described. The transmitting / receiving unit 11a is mainly realized by the processing of the CPU 101 shown in FIG. 4 and the network I / F 105, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 100. The transmitting / receiving unit 11a also plays a role as an initiating unit, and performs processing to start communication with other communication terminals such as the electronic whiteboard 1b. The transmitting / receiving unit 11a (the transmitting / receiving unit 11b) functions as an example of a receiving means in this embodiment. The receiving unit 12a is mainly realized by the processing of the CPU 101 shown in FIG. 4, the electronic pen controller 116, the contact sensor 114, and the sensor controller 115, and receives various inputs from the user using the electronic pen 190 or the like, and various inputs using a part of the user's body such as the hand. The electronic pen 190 is an example of a predetermined tool in this embodiment. The predetermined tool may be any type of tool that has an interface that can communicate with the electronic pen controller 116 shown in FIG. 4 instead of the electronic pen 190. Moreover, the reception unit 12a (the reception unit 12b) functions as an example of a reception unit in this embodiment.

[0051] The video / audio processor 13a is mainly realized by the processing of the CPU 101, the microphone 140, the speaker 150, and the camera 160 shown in Fig. 4, and performs main processing of the video conference function. For example, the video / audio processor 13a performs digital processing such as encoding of video data and audio data based on the output signal of the microphone 140 and the output signal of the camera 160. The video / audio processor 13a also generates a video signal and an audio signal based on the video data and audio data received by the transmitting / receiving unit 11a. Furthermore, the video / audio processor 13a performs processing to combine video data with different resolutions.

[0052] The display control unit 14a is mainly realized by the processing of the CPU 101 and the display controller 113 shown in Fig. 4, and performs control for outputting a video signal (image signal) and the like to the display 180. In this embodiment, the display control unit 14a (display control unit 14b) functions as an example of a display control means. Furthermore, the display 180 functions as an example of a display means.

[0053] The determination unit 15a is mainly realized by the processing of the CPU 101 shown in FIG. 4, and performs various determinations.

[0054] The measurement unit 16a is mainly realized by the processing of the CPU 101, the contact sensor 114, and the sensor controller 115 shown in FIG. 4, and measures the time during which the electronic pen 190, the hand H, or the like is in contact with the display screen of the display 180 of the electronic whiteboard 1a, and the length (distance) of an input image drawn and input by the electronic pen 190, the hand H, or the like. Specifically, the measurement unit 16a measures the time during which the electronic pen 190, the hand H, or the like is in contact with the display screen of the display 180 of the electronic whiteboard 1a, using clock information or the like used by the CPU 101. Furthermore, the measurement unit 16a measures the length (distance) of an input image drawn and input by the electronic pen 190, the hand H, or the like, using various sensors such as the contact sensor 114. The measurement unit 16a (measurement unit 16b) functions as an example of a measurement means in this embodiment. In this embodiment, a stroke image input by the electronic pen 190, or the like, is treated as an example of an input image.

[0055] The image processing unit 17a is mainly realized by the processing of the CPU 101 shown in FIG. 4, and performs the main processing of the electronic whiteboard function. For example, the image processing unit 17a creates a stroke image and stroke data based on strokes of the electronic pen 190 or the like received by the reception unit 12a. Also, for example, the image processing unit 17a creates a stroke image based on stroke data received by the transmission / reception unit 11a. Also, the image processing unit 17a generates an image signal indicating a document image based on image data of the document image received by the transmission / reception unit 11a. Also, the image processing unit 17a captures a screen (screen capture; hereinafter, simply referred to as "screen capture") in response to a screen capture process (screen capture process) by a user. That is, the image processing unit 17a (image processing unit 17b) functions as an example of a capture means in this embodiment. Furthermore, the image processing unit 17a performs the main processing of the electronic whiteboard function. For example, the image processing unit 17a performs superimposition processing (composite processing) of image layers as shown in FIG. 3 described above, and the like.

[0056] The providing / obtaining unit 18a is mainly realized by the processing of the CPU 101 and the external device connection I / F 208 shown in FIG. 4, and provides and obtains data through communication with an external device (for example, PC 2).

[0057] The storage / reading processing unit 19a is mainly realized by the processing of the CPU 101 shown in FIG. 4, the external device connection I / F 106, the ROM 102, and the SSD 104, and performs processing such as storing various data in the memory unit 1000a or a recording medium 1010a such as a USB memory 130, and reading out various data stored in the memory unit 1000a or the recording medium 1010a.

[0058] Furthermore, video data and audio data received during communication with other terminals are overwritten and stored in the storage unit 1000a every time they are received. Of these, images or videos are displayed on the display 180 by the video data before being overwritten, and audio is output from the speaker 150 by the audio data before being overwritten.

[0059] <Functional configuration of the electronic blackboard 1b> As shown in FIG. 7, the electronic blackboard 1b includes a transmission / reception unit 11b, a reception unit 12b, a video / audio processing unit 13b, a display control unit 14b, a determination unit 15b, a measurement unit 16b, an image processing unit 17b, a provision / acquisition unit 18b, and a storage / readout processing unit 19b. Further, the electronic blackboard 1b has a storage unit 1000b constructed by the RAM 103 and the SSD 104 shown in FIG. 4. Since the transmission / reception unit 11b, the reception unit 12b, the video / audio processing unit 13b, the display control unit 14b, the determination unit 15b, the image processing unit 17b, the provision / acquisition unit 18b, and the storage / readout processing unit 19b each have the same functions as the transmission / reception unit 11a, the reception unit 12a, the video / audio processing unit 13a, the display control unit 14a, the determination unit 15a, the image processing unit 17a, the provision / acquisition unit 18a, and the storage / readout processing unit 19a, the descriptions thereof are omitted. Similarly, since the storage unit 1000b has the same functions as the storage unit 1000a, the description thereof is omitted.

[0060] <Functional configuration of the PC 2a> As shown in FIG. 7, the PC 2a includes a transmission / reception unit 21a, a display control unit 24a, a provision / acquisition unit 28a, and a storage / readout processing unit 29a. Each of these units is a function or means realized by any of the components shown in FIG. 5 operating according to an instruction from the CPU 201 in accordance with an information processing program developed from the HD 204 onto the RAM 203. Further, the PC 2a has a storage unit 2000 constructed by the RAM 203 and the HD 204 shown in FIG. 5.

[0061] <<Functional configurations of the PC 2a>> Next, each functional configuration of the PC 2a will be described in detail. The transmission / reception unit 21a of the PC 2a shown in Fig. 7 is mainly realized by the processing of the CPU 201, the external device connection I / F 208, and the network I / F 209 shown in Fig. 5, and transmits and receives various data (or information) to and from other terminals, devices, or systems via the communication network 100. The display control unit 24a is mainly realized by the processing of the CPU 201 and the display 206 shown in Fig. 5, and controls the output of a video signal (image signal) to the display 206. The providing / acquiring unit 28a is mainly realized by the processing of the CPU 201 and the external device connection I / F 208 shown in Fig. 5, and provides and acquires data by communication with an external device (for example, the electronic whiteboard 1). 5, and performs processing such as storing various data in the storage unit 2000 and reading out various data stored in the storage unit 2000. Note that the PC 2b that communicates with the electronic whiteboard 1b has the same functions as the PC 2a, and therefore the description thereof will be omitted.

[0062] <Functional configuration of relay device> 7, the relay device 3 has a transmitting / receiving unit 31 also serving as a forwarding unit, a determining unit 35, and a storage / reading processing unit 39. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 5 operates in response to an instruction from the CPU 501 in accordance with the relay program loaded from the HD 504 onto the RAM 503. The relay device 3 also has a storage unit 3000 constructed by the RAM 303 and HD 304 shown in Fig. 5.

[0063] <<Functional configuration of relay device>> Next, each functional configuration of the relay device 3 will be described in detail. The transmission / reception unit 31 of the relay device 3 shown in Fig. 7 is mainly realized by the processing of the CPU 301, the external device I / F 308, and the network I / F 309 shown in Fig. 5, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 100. The transmission / reception unit 31 also plays a role as a transfer unit, and transfers video data and audio data transmitted from a predetermined terminal to another terminal. The determination unit 35 performs various determinations such as determining the delay state of data.

[0064] The storage / readout processing unit 39 is mainly realized by the processing of the CPU 301, the ROM 302, the HDD controller 305, and the HD 304 shown in FIG. 5, and performs processing such as storing various data in the memory unit 3000 and reading out various data stored in the memory unit 3000.

[0065] <Functional configuration of the communication management system> 7, the communication management system 5 has a transmission / reception unit 51, an authentication unit 52, a generation unit 53, a selection unit 54, a judgment unit 55, and a storage / readout processing unit 59. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 5 operates in response to an instruction from the CPU 501 in accordance with the communication management program loaded from the HD 504 onto the RAM 503. The communication management system 5 also has a storage unit 5000 constructed by the RAM 503 and HD 504 shown in Fig. 5.

[0066] (Authentication Management Table) FIG. 8 is a conceptual diagram showing an example of an authentication management table. In the storage unit 5000, an authentication management DB 5001 configured by the authentication management table shown in FIG. 8 is constructed. In this authentication management table, each password is associated with each terminal ID of all the electronic whiteboards 1 managed by the communication management system 5 and managed. For example, the authentication management table shown in FIG. 7 indicates that the terminal ID of the electronic whiteboard 1a (communication terminal) is "01aa" and the password is "aaaa". Note that the password is an example of authentication information, and the authentication information also includes an access token. In this embodiment, the authentication management table (authentication management DB 5001) functions as an example of an authentication management means.

[0067] (Terminal Management Table) FIG. 9 is a conceptual diagram showing an example of a terminal management table. In the storage unit 5000, a terminal management DB 5002 configured by the terminal management table shown in FIG. 9 is constructed. In this terminal management table, for each terminal ID for identifying each electronic whiteboard 1 (communication terminal), the destination name when each electronic whiteboard 1 is the destination, the operation status of each electronic whiteboard 1, the reception date and time when the communication management system 5 receives login request information described later, and the IP address of each electronic whiteboard 1 (communication terminal) are associated and managed. For example, in the terminal management table shown in FIG. 9, it is shown that the electronic whiteboard 1a with the terminal ID "01aa" has the terminal name "Japan Tokyo Office AA Terminal", the operation status "Online (communication possible)", the date and time when the communication management system 5 receives the login request information is "April 10, 2015, 13:40", and the IP address of this terminal 1aa is "1.2.1.3". The terminal ID, the destination name, and the IP address of the terminal are stored when each electronic whiteboard 1 is pre-registered to receive services provided by the communication management system 5. In this embodiment, the terminal management table (terminal management DB 5002) functions as an example of a terminal management means.

[0068] (Destination list management table) FIG. 10 is a conceptual diagram showing an example of a destination list management table. In the storage unit 5000, a destination list management DB 5003 configured by the destination list management table shown in FIG. 10 is constructed. In this destination list management table, all terminal IDs of destination terminals registered as candidates of the electronic whiteboard 1 (destination terminal) are associated and managed with the terminal ID of the electronic whiteboard 1 (start terminal) requesting the start of communication. For example, in the destination list management table shown in FIG. 10, it is shown that the candidate of the destination terminal that can request the start of communication from the start terminal (electronic whiteboard 1a) with the terminal ID "01aa" is the electronic whiteboard 1b with the terminal ID "01ba". The candidate of the destination terminal is updated by being added or deleted in response to an addition or deletion request from any start terminal to the communication management system 5.

[0069] The destination list is an example of destination information, and the destination information may not be in a list format, but may instead list information about destinations such as terminal IDs. In this embodiment, the destination list management table (destination list management DB 5003) functions as an example of a destination list management means.

[0070] (Session management table) Fig. 11 is a conceptual diagram showing an example of a session management table. In the storage unit 5000, a session management DB 5004 configured by the session management table shown in Fig. 11 is constructed. In this session management table, for each communication session ID for identifying a session for mutual communication between each electronic whiteboard (communication terminal) and the relay device 3, the relay device ID of the relay device 3 used, the terminal ID of the electronic whiteboard 1 (start terminal), the terminal ID of the electronic whiteboard 1 (destination terminal), the reception delay time (ms) when the video data is received at the destination terminal, and the reception date and time when the communication management system 5 receives the delay information indicating this delay time after the delay information is transmitted from the destination terminal are associated and managed. 11, in a communication session executed using session ID "se01", a relay device (relay device ID "111a") relays video data and audio data between an electronic whiteboard with terminal ID "01aa" and an electronic whiteboard with terminal ID "01db", and the delay time of the video data at electronic whiteboard 1 (destination terminal) at "13:41 on June 19, 2020" is 200 (ms). In this embodiment, the session management table (session management DB 5004) functions as an example of a session management means.

[0071] (Relay device management table) FIG. 12 is a conceptual diagram showing an example of a relay device management table. In the storage unit 5000, a relay device management DB 5005 configured by the relay device management table shown in FIG. 12 is constructed. In this relay device management table, the operation state of each relay device 3, the reception date and time when the state information indicating the operation state was received by the communication management system 5, the IP address of the relay device 3, and the maximum data transmission rate (Mbps) of the relay device 3 are associated and managed for each relay device ID. For example, in the relay device management table shown in FIG. 12, it is shown that the operation state of the relay device 3 with the relay device ID "111a" is "online", the date and time when the state information was received by the communication management system 5 is "13:00 on April 10, 2019", the IP address of this relay device 3 is "1.2.1.2", and the maximum data transmission rate of this relay device 3 is 100 Mbps. In this embodiment, the relay device management table (relay device management DB 5005) functions as an example of a relay device management means.

[0072] <<Functional configuration of the communication management system>> Next, a detailed description will be given of each functional configuration of the communication management system 5. The transmission / reception unit 51 of the communication management system 5 shown in Fig. 7 is mainly realized by the processing of the CPU 501, the external device I / F 508, and the network I / F 509 shown in Fig. 5, and transmits and receives various data (or information) to and from other terminals, devices, or systems via the communication network 100.

[0073] The authentication unit 52 is mainly realized by the processing of the CPU 501 shown in FIG. 5, and searches the authentication management DB 5001 of the storage unit 5000 using the terminal ID and password included in the login request received via the transmission / reception unit 51 as search keys, and performs authentication by determining whether the same set of terminal ID and password is managed in the authentication management DB 5001. The generation unit 53 is mainly realized by the processing of the CPU 501 shown in FIG. 5, and generates a session ID for identifying a communication session based on a communication start request from the electronic whiteboard 1 (see step S62). The selection unit 54 is mainly realized by the processing of the CPU 501 shown in FIG. 5, and performs a process of finally selecting one relay device 3 from multiple relay devices 3. The judgment unit 55 is mainly realized by the processing of the CPU 501 shown in FIG. 5, and performs various judgments in the communication management system 5. The storage / readout processing unit 59 is mainly realized by the processing of the CPU 501, the ROM 502, the HDD controller 505, and the HD 504 shown in FIG. 5, and performs processing such as storing various data in the memory unit 5000 and reading out various data stored in the memory unit 5000.

[0074] <Functional configuration of the image storage device> As shown in Fig. 7, the image storage device 7 has a transmission / reception unit 71, and a storage / readout processing unit 79. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 5 operates in response to an instruction from the CPU 701 in accordance with an image storage program loaded from the HD 704 onto the RAM 703. The image storage device 7 also has a memory unit 7000 constructed by the RAM 703 and HD 704 shown in Fig. 5.

[0075] <<Functional configuration of the image storage device>> Next, each functional configuration of the image storage device 7 will be described in detail. The transmission / reception unit 71 of the image storage device 7 shown in Fig. 7 is mainly realized by the processing of the CPU 701, the external device I / F 708, and the network I / F 709 shown in Fig. 5, and transmits and receives various data (or information) to and from other terminals, devices, or systems via the communication network 100. The storage / readout processing unit 79 is mainly realized by the processing of the CPU 701, the ROM 702, the HDD controller 705, and the HD 704 shown in Fig. 5, and performs processing to store various data in the storage unit 7000 and to read out various data stored in the storage unit 7000.

[0076] [Processing or Operation of the Embodiment] Next, the processing or operation in the communication system according to this embodiment will be described with reference to FIGS.

[0077] <Preparation for remote communication> First, the preparation process for communication performed by the electronic whiteboard 1a as a login request terminal will be described with reference to Fig. 13 and Fig. 14. Fig. 13 is a sequence diagram showing an example of the process in the preparation stage for starting remote communication between electronic whiteboards. Fig. 14 is a diagram showing an example of a destination list screen displayed on the electronic whiteboard 1a. The process of login sharing performed by the electronic whiteboard 1b is similar to that performed by the electronic whiteboard 1a, and therefore will not be described.

[0078] First, when the power switch 122 of the interactive whiteboard 1a is turned on, the reception unit 12a receives the power-on command (step S22).

[0079] Next, the transmitting / receiving unit 11a transmits login request information indicating a request for login authentication to the communication management system 5 via the communication network 100 (step S23). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the login request information transmitted from the transmitting / receiving unit 11a of the electronic whiteboard 1a. The login request information includes the terminal ID and password of the electronic whiteboard 1a.

[0080] Next, the authentication unit 52 of the communication management system 5 searches the authentication management table (see FIG. 8) using the terminal ID and password included in the login request information received via the transmission / reception unit 51 as search keys, and determines whether the same terminal ID and the same password are managed, thereby authenticating the terminal (step S24). Here, the description will continue assuming that the same terminal ID and the same password are managed by the storage / readout processing unit 59.

[0081] If the authentication unit 52 determines that the login request is from a terminal with valid usage rights, assuming that the same terminal ID and the same password are managed, the storage / readout processing unit 59 changes the operation status field of the record of the terminal ID received in step S23 to "online (communication possible)" in the terminal management table (see FIG. 9), and stores the reception date and time when the login request information was received in step S23 in the reception date and time field (step S25). As a result, the terminal management table manages the terminal ID "01aa" in association with the operation status "online (communication possible)", the reception date and time "2020.6.19.13:40", and the IP address "1.2.1.3". Note that the IP address of the terminal may not be registered in advance, but may be transmitted from the electronic whiteboard 1a in step S23.

[0082] Next, the storage / read processing unit 59 adds a new record including the terminal ID of the interactive whiteboard 1a received in step S23 to the session management table (see FIG. 11) and manages the new record (step S26).

[0083] Then, the transmitting / receiving unit 51 of the communication management system 5 transmits authentication result information indicating the authentication result obtained by the processing of step 24 to the electronic whiteboard 1a that made the login request via the communication network 100 (step S27). As a result, the transmitting / receiving unit 11a of the electronic whiteboard 1a receives the authentication result information transmitted from the transmitting / receiving unit 51 of the communication management system 5.

[0084] When the transmitting / receiving unit 11a of the login request terminal (electronic whiteboard 1a) receives authentication result information indicating that the terminal has been determined to have valid usage authority, the transmitting / receiving unit 11a transmits destination list request information indicating a request for a destination list to the communication management system 5 via the communication network 100 (step S28). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the destination list request information transmitted from the transmitting / receiving unit 11a of the electronic whiteboard 1a.

[0085] Next, the storage / read processing unit 59 searches the destination list management table (see FIG. 10) using the terminal ID "01aa" of the login request terminal (electronic whiteboard 1a) as a search key to read out the terminal IDs of destination candidates capable of communicating with the login request terminal (electronic whiteboard 1a) and reads out the destination names corresponding to these terminal IDs from the terminal management table (see FIG. 9) (step S29). Here, the terminal IDs of the destination candidates corresponding to the terminal ID "01aa" of the login request terminal (electronic whiteboard 1a) and the destination names corresponding to these are extracted.

[0086] Next, the transmission / reception unit 51 of the communication management system 5 reads out the data of the destination list frame and the data of the icon indicating the operating status from the memory unit 5000 via the memory / read processing unit 59 (step S30), and transmits the "destination list information (destination list frame, icon, terminal ID, destination name)" including the destination list frame and icon, as well as the terminal ID and destination name read out by the memory / read processing unit 59 to the login request terminal (electronic whiteboard 1a) (step S31).

[0087] As a result, in the login request terminal (electronic whiteboard 1a), the transmitter / receiver 11a receives the destination list information transmitted from the transmitter / receiver 51 of the communication management system 5, and the storage / read processing unit 19a stores the destination list information in the memory unit 1000a (step S32).

[0088] As described above, in this embodiment, the destination list information is not managed by each terminal, but the communication management system 5 centrally manages the destination list information of all terminals. Therefore, even in the following cases, the communication management system 5 handles all of the cases at once, so it is possible to save the trouble of changing the destination list information on each terminal. Here, the following cases refer to cases where a new electronic whiteboard is included in the communication system, where a new model of terminal is included in place of a terminal already included, where the appearance of the destination list frame is changed, etc.

[0089] In addition, the storage / read processing unit 59 of the communication management system 5 searches the terminal management table (see Figure 9) using the terminal IDs of the destination candidates read out above as a search key, and obtains each operating status of the electronic whiteboard as a destination candidate by reading out the corresponding operating status for each of the terminal IDs above (step S33).

[0090] Next, the transmitting / receiving unit 51 transmits the terminal ID used as the search key in step S33 above and the "terminal status information" including the operation status of each corresponding destination terminal to the login request terminal (electronic whiteboard 1a) via the communication network 100 (step S34). As a result, the transmitting / receiving unit 11a of the electronic whiteboard 1a receives the "terminal status information" transmitted from the transmitting / receiving unit 51 of the communication management system 5.

[0091] Next, the memory / read processing unit 19a of the login sharing terminal (electronic whiteboard 1a) sequentially stores the terminal status information received from the communication management system 5 in the memory unit 1000a (step S35). Thus, by receiving the status information of each of the electronic whiteboards, the login request terminal (electronic whiteboard 1a) can obtain the current operating status of each of the electronic whiteboards 1b and the like that are potential destinations that can communicate with the login request terminal (electronic whiteboard 1a).

[0092] Next, the display control unit 14a of the login request terminal (electronic whiteboard 1a) creates a destination list reflecting the status of the terminal as a destination candidate based on the destination list information and the terminal status information stored in the storage unit 1000a, and the display control unit 14a displays the destination list screen 1100 shown in Fig. 14 on the display 180 of the electronic whiteboard 1a (step S36). On this destination list screen 1100, an icon indicating the operating status, a terminal ID, and a destination name are displayed for each destination candidate. In Fig. 14, the icons indicating the operating status of each terminal are displayed as "offline" and "online (communication available)" from the top.

[0093] Meanwhile, the storage / readout processing unit 59 of the communication management system 5 searches the destination list management table (see FIG. 10) based on the terminal ID "01aa" of the login request terminal (electronic whiteboard 1a) to extract the terminal IDs of other terminals that have registered the terminal ID "01aa" of the login request terminal (electronic whiteboard 1a) as a destination candidate (step S37). In the destination list management table shown in FIG. 10, the terminal IDs of the other terminals that are read out are "01ba", "01ca", "01da", etc.

[0094] Next, the memory / read processing unit 59 of the communication management system 5 searches the terminal management table (see FIG. 9) based on the terminal ID “01aa” of the login request terminal (electronic whiteboard 1a) and obtains the operating status of the login request terminal (electronic whiteboard 1a) (step S38).

[0095] Then, the transmitting / receiving unit 51 transmits "terminal status information" including the terminal ID "01aa" of the login request terminal (electronic whiteboard 1a) acquired in step S38 and the operating status "online" to the terminals related to the terminal IDs extracted in step S37 and whose operating status is "online" in the terminal management table (see FIG. 9) (step S39). When transmitting the terminal status information to the electronic whiteboard 1b, the transmitting / receiving unit 51 refers to the IP address of the electronic whiteboard managed in the terminal management table (see FIG. 9) based on each terminal ID.

[0096] This allows the terminal ID "01aa" and the operating status "online" of the login request terminal (electronic whiteboard 1a) to be communicated to each of the other destination terminals that can communicate with the login request terminal (electronic whiteboard 1a) as a destination candidate. Therefore, the destination candidate (electronic whiteboard 1b, etc.) can also display the status of the destination candidate (step S40).

[0097] <Remote communication start process> Next, a process in which the electronic whiteboard 1a starts remote communication with the electronic whiteboard 1b will be described with reference to Fig. 15. Fig. 15 is a sequence diagram showing an example of a process in which remote communication is started.

[0098] First, when a user of the request source terminal (electronic whiteboard 1a) operates a destination candidate (terminal ID "01ba") shown in FIG. 9 to select the electronic whiteboard 1b, the reception unit 12a shown in FIG. 7 receives a request to start a call with the destination terminal (electronic whiteboard 1b) (step S61). Then, the transmission / reception unit 11a of the request source terminal (electronic whiteboard 1a) transmits start request information indicating a desire to start a call to the communication management system 5 (step S62). As a result, the transmission / reception unit 51 of the communication management system 5 receives the start request information and also receives the IP address of the request source terminal (electronic whiteboard 1a) that is the sender. The start request information includes the terminal ID "01aa" of the request source terminal (electronic whiteboard 1a) and the terminal ID "01ba" of the destination terminal (electronic whiteboard 1b).

[0099] Then, based on the terminal ID "01aa" of the request source terminal (electronic whiteboard 1a) and the terminal ID "01ba" of the destination terminal (electronic whiteboard 1b) included in the start request information, the storage / read processing unit 59 changes the operation status field portions of the records including the terminal ID "01aa" and the terminal ID "01ba" in the terminal management table (see FIG. 9) to "busy" (step S63). Note that in this state, the request source terminal (electronic whiteboard 1a) and the destination terminal (electronic whiteboard 1b) have not started a video conference, but are in a busy state, and when a third electronic whiteboard terminal tries to talk to the request source terminal (electronic whiteboard 1a) or the destination terminal (electronic whiteboard 1b), a notification sound or display indicating a so-called busy state is output.

[0100] Next, a process of executing a session for selecting a relay device 3 to be actually used will be described. First, the generation unit 53 of the communication management system 5 generates a session ID used for executing a session for selecting a relay device 3 (step S64). Here, a case where a session ID "se01" is generated will be described.

[0101] Then, the storage / read processing unit 59 stores and manages the session ID "se01" generated in step S64 above, the terminal ID "01aa" of the requesting terminal (electronic whiteboard 1a), and the terminal ID "01ba" of the destination terminal (electronic whiteboard 1b) in a session management table (see FIG. 11) in association with each other (step S65).

[0102] Next, the selection unit 54 of the communication management system 5 shown in Fig. 7 selects a relay device 3 for relaying the call between the two points of the request source terminal (electronic whiteboard 1a) and the destination terminal (electronic whiteboard 1b) based on the relay device management table (see Fig. 12) and the terminal management table (see Fig. 9) (step S66). Specifically, among the relay device IDs related to relay devices whose operating status is "online" in the relay device management table (see Fig. 12), the relay device ID related to the relay device 3 whose IP address is close to the IP address of the request source terminal (electronic whiteboard 1a) in the terminal management table (see Fig. 9) is selected. Here, the case where the relay device 30a (relay device ID "111a") is selected will be described below.

[0103] When the relay device selection process in step S66 is completed, the transmitter / receiver 51 of the communication management system 5 transmits relay device selection information to the requesting terminal (electronic whiteboard 1a) (step S67-1). This enables the requesting terminal (electronic whiteboard 1a) to receive and acquire the IP address of the communication management system 5 that is the sender of the relay device selection information. The relay device selection information includes the IP address of the relay device 3 selected in step S66 and the session ID "se01" generated in step S64.

[0104] Furthermore, the transmitting / receiving unit 51 of the communication management system 5 transmits relay device selection information to the destination terminal (electronic whiteboard 1b) (step S67-2). This allows the destination terminal (electronic whiteboard 1b) to obtain the IP address of the communication management system 5 that is the sender of the relay device selection information in the execution of the session with the session ID "se01". The relay device selection information includes the IP address of the relay device 3 selected in the above step S66, the terminal ID "01aa" of the request source terminal (electronic whiteboard 1a), and the session ID "se01" generated in the above step S64.

[0105] Next, in response to the processing of step S67-1, the transmitting / receiving unit 11 of the request source terminal (electronic whiteboard 1a) transmits to the communication management system 5 reception completion information indicating that reception of the relay device selection information has been completed by the processing of step S67-1 (step S68-1). As a result, the communication management system 5 receives and acquires information indicating that transmission of the relay device selection information being executed in the specific session ID "se01" has been completed. The reception completion information includes the session ID transmitted and received in the processing of step S67-1.

[0106] Furthermore, in response to the process of step S67-2, the destination terminal (electronic whiteboard 1b) similarly transmits to the communication management system 5 reception completion information indicating that reception of the relay device selection information has been completed by the process of step S67-2 (step S68-2). In this case as well, the communication management system 5 obtains the information that transmission of the relay device selection information being executed in the specific session ID "se01" has been completed.

[0107] As described above, the interactive whiteboards 1a and 1b can conduct a video conference by transmitting and receiving video data and audio data via the relay device 3 selected in step S66.

[0108] <Communication processing in information and communication system 8> Here, the communication process between PC 2 and the electronic whiteboard 1 will be described. Fig. 16 is a sequence diagram showing an example of the process between PC 2 and the electronic whiteboard 1. First, PC 2a and electronic whiteboard 1a (PC 2b and electronic whiteboard 1b) constituting an information communication system 8 are connected to each other via, for example, a VGA cable, an HDMI cable, or the like (step S81). At this time, in the information communication system 8, initial settings including establishment of communication based on each communication standard such as VGA or HDMI are performed. Since the initial settings including establishment of communication are well-known techniques, a description thereof will be omitted here.

[0109] Next, the display control unit 24a of the PC 2a displays the document image (moving image) on the display 206 (step S82-1). In response to this, the display control unit 14a of the electronic whiteboard 1a displays selection buttons (or icons, hereinafter simply referred to as "selection buttons") at a predetermined position on the display 180 of the electronic whiteboard 1a (step S82-2). The selection buttons are the "share" button bs1, the "PC" button bc1, the "pen" button bp1, the "marker" button bm1, and the "camera" button ba1. Of these buttons, the "share" button b1 is a button operated when sharing the document image d1 of the site B. The "PC" button bs1 is a button operated when selecting whether the display screen is a PC screen or an imported screen. The "pen" button bp1 is a button operated when selecting a pen mode in which a stroke image or the like is input using the electronic pen 190, for example. The "marker" button bm1 is a button operated when selecting a marker mode in which, for example, a hand H is traced on the surface of the display 180 and the traced trajectory is displayed as a marker (for example, a thick colored line, also called a "pointer"). The "camera" button ba1 is a button operated when capturing (taking a screen capture) a screen displayed on the display 180, and is an example of a capture selection means in this embodiment. The "share" button bs1, the "PC" button bc1, the "pen" button bp1, the "marker" button bm1, and the "camera" button ba1 are displayed side by side on the right side of the display 180. Note that each of the selection buttons described above may be configured with each selection icon. Also, depending on the model of the electronic whiteboard, the function of the "pen" button bp1 and the function of the "marker" button bm1 may be provided as a button (icon) that integrates the functions of the "pen" button bp1 and the "marker" button bm1 into one button.

[0110] Next, the transmitting / receiving unit 11a of the PC 2a transmits image data representing the material image (moving image) displayed in step S82-1 to the electronic whiteboard 1a (step S83). As a result, the transmitting / receiving unit 11a of the electronic whiteboard 1a receives the image data representing the material image (moving image) transmitted by the PC 2a.

[0111] Next, the display control unit 14a displays the material image (moving image) displayed on the PC 2 on the display 180 based on the received image data representing the material image (moving image) (step S84).

[0112] Next, the accepting unit 12a accepts an operation from the user on a selection button displayed on the display 180 (step S85).

[0113] Furthermore, the electronic whiteboard 1a proceeds to step S101, which will be described later, and exits this flow (step S86).

[0114] In the information communication system 8 according to the present embodiment, when the process of step S83 described above is executed, other devices may be present between the PC 2 (information processing device) and the electronic whiteboard 1 (communication terminal). In other words, each piece of information transmitted and received between the PC 2 and the electronic whiteboard 1 may be configured to be transmitted and received once via another device.

[0115] <Communication processing of document images> Next, a communication process of the document image displayed on the electronic whiteboard 1 and the input stroke image will be described with reference to Figs. 17 to 19. Fig. 17 is a sequence diagram showing an example of a process of sharing image data of the document image and stroke data of the stroke image. Fig. 18 is a sequence diagram showing an example of an upload process and a download process of the image data of the document image. Here, a case where the image data of the document image and the stroke data of the stroke image are shared between the electronic whiteboards 1a and 1b, and the document image displayed on the electronic whiteboard 1a and the input stroke image are also displayed on the electronic whiteboard 1b will be described. Furthermore, Fig. 19 is an example of a screen displayed on each electronic whiteboard in each process of Figs. 17 and 18. In Fig. 19, (A) is an example of a screen of the electronic whiteboard 1a, (B) is an example of a screen of the electronic whiteboard 1b, (C) is an example of a screen of the electronic whiteboard 1a, and (D) is an example of a screen of the electronic whiteboard 1b.

[0116] First, in the electronic whiteboard 1a at the site A, the display control unit 14a displays the screen shown in FIG. 19(A) on the display 180. The display contents are the document image c1, the background image d1, the video v2, and the above-mentioned "Share" button bs1, the "PC" button bc1, the "Pen" button bp1, the "Marker" button bm1, and the "Camera" button ba1. The document image c1 is, for example, an image shown as a pie chart. The background image d1 is, for example, a character string such as "Ri-Photo Electronic Whiteboard". The video v2 is, for example, an image of two people at the site B sitting facing each other at a table. In this state, when the user A1 of the electronic whiteboard 1a performs an operation such as pressing or tapping the "Share" button b1, the reception unit 12a receives a sharing process of the document image (video) from the user A1 (step S101). In the screen example shown in FIG. 19, the "Share" button bs1 is, for example, highlighted or the outer frame of the button is displayed in a thick color. Depending on the model of the electronic whiteboard, the "PC" button bs1 and the "marker" button bm1 may be selectable separately as described above, or one of the buttons may not be supported.

[0117] Next, the transmitting / receiving unit 11a transmits request information indicating a request for a URL (Uniform Resource Locator) indicating the storage location of the image data of the document image to the communication management system 5 (step S102). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request information. Note that the URL is an example of storage location information, and the storage location information also includes a URI (Uniform Resource Identifier).

[0118] Next, the generating unit 53 of the communication management system 5 generates a URL indicating the storage location of the image data of the document image (step S103).

[0119] Then, the transmitting / receiving unit 51 transmits the URL of the image data generated by the generating unit 53 to the interactive whiteboard 1a (step S104). As a result, the transmitting / receiving unit 11a of the interactive whiteboard 1a receives the URL of the image data.

[0120] Next, the transmitting / receiving unit 11a of the electronic whiteboard 1a transmits a notification of the start of uploading image data of the material image to the communication management system 5 (step S105). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the start notification.

[0121] Next, the transmitting / receiving unit 51 transfers the start notification to the remote conference partner electronic whiteboard 1b (step S106). As a result, the transmitting / receiving unit 11b of the electronic whiteboard 1b receives the start notification.

[0122] In the electronic whiteboard 1b at the site B, the display control unit 14b displays the screen shown in FIG. 19(B) on the display 180 (step S107). Here, the "share" button bs2, the "PC" button bc2, the "pen" button bp2, the "marker" button bm2, and the "camera" button ba2, which are operated when sharing the video v1 at the site A, are displayed at the same positions as those displayed on the display 180 of the electronic whiteboard 1a. In this state, the display control unit 14b outputs an hourglass icon c2, which visually indicates the time lapse of downloading the image data of the document image, in response to the start notification in step S106. The "share" button bs2, the "PC" button bc2, the "pen" button bp2, the "marker" button bm2, and the "camera" button ba2 have the same functions as the "share" button bs1, the "PC" button bc1, the "pen" button bp1, the "marker" button bm1, and the "camera" button ba1 on the electronic whiteboard 1a.

[0123] The passage of time can be indicated by a still hourglass or an animated hourglass with moving sand. The hourglass icon c2 is an example of schedule information indicating that image data is scheduled to be downloaded. Other examples of schedule information may be icons other than an hourglass (for example, a clock icon), or may be any one of letters and symbols instead of an icon, or a combination of an icon, letters, and symbols. Schedule information may also be a notification by sound. In this case, sound is output from speaker 150 by video / audio processor 13a, not display controller 14a.

[0124] The electronic whiteboard 1a may perform the process of step S102 after the process of step S105. The communication management system 5 may transmit the URL of the image data in step S104 after receiving the notification of the start of uploading the image data in step S105.

[0125] Next, the transmitting / receiving unit 11a of the electronic whiteboard 1a uploads the image data of the document image to the URL related to the image storage device 7 received in step S104 (step S108). As a result, the transmitting / receiving unit 71 of the image storage device 7 receives the image data. Then, the storage / read processing unit 79 of the image storage device 7 stores the image data for the URL in the storage unit 7000 (step S109).

[0126] <Stroke image communication processing> Next, at the site A, the user A1 touches and moves the electronic pen 190 or the hand H on the display 180 of the electronic whiteboard 1a, and the reception unit 12a receives the input of the movement stroke (step S110). In this embodiment, the movement stroke is used as a trajectory drawn by the electronic pen 190 or the hand H.

[0127] Next, the image processing unit 17a creates stroke data (for example, coordinate data (x, y)) for displaying a stroke image on the two-dimensional display 180 based on the stroke processing performed by the user A1. At the same time, the display control unit 14a displays the stroke image on the display 180 of the electronic whiteboard 1a (step S111). As a result, as shown in FIG. 19(C), a stroke image st1 represented by a curved arrow is displayed. Furthermore, in FIG. 19(C), similarly to FIG. 19(A), the document image c1, background image d1, video v2, as well as the "share" button bs1, "PC" button bc1, "pen" button bp1, "marker" button bm1, and "camera" button ba1 are each displayed on the display 180 of the electronic whiteboard 1a.

[0128] Next, the transmitting / receiving unit 11a transmits stroke data for reproducing the stroke image created in step S111 to the communication management system 5 (step S112). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the stroke data of the stroke image transmitted from the electronic whiteboard 1a.

[0129] Then, in the communication management system 5, the storage / read processing unit 59 stores the stroke data (step S113).

[0130] Next, the transmitting / receiving unit 51 transfers the stroke data to the partner electronic whiteboard 1b (step S114). As a result, the transmitting / receiving unit 11b of the electronic whiteboard 1b receives the stroke data transmitted from the communication management system 5.

[0131] Then, in the electronic whiteboard 1b, the image processor 17b creates a stroke image based on the stroke data, and the display controller 14b displays the stroke image on the display 180 of the electronic whiteboard 1b (step S115). As a result, as shown in FIG. 19(D), a stroke image st2 represented by a curved arrow, which is the same as the stroke image st1 displayed on the electronic whiteboard 1a of the other party (start terminal), is displayed. Furthermore, in FIG. 19(D), the document image C1, background image d2, video v1, as well as the "share" button bs2, "PC" button bc2, "pen" button bp2, "marker" button bm2, and "camera" button ba2 are displayed on the display 180 of the electronic whiteboard 1b.

[0132] 18, the transmitting / receiving unit 71 of the image storage device 7 transmits a completion notification indicating that the upload of the image data of the document image is completed to the electronic whiteboard 1a (step S121). As a result, the transmitting / receiving unit 11a of the electronic whiteboard 1a receives the completion notification transmitted from the image storage device 7.

[0133] Next, the transmitting / receiving unit 11a of the electronic whiteboard 1a transmits a completion notification indicating that the upload of the image data of the document image is completed to the communication management system 5 (step S122). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the completion notification. The completion notification includes the URL of the image data received in step S104.

[0134] Then, the transmitting / receiving unit 51 transfers the completion notification including the URL to the other party's electronic whiteboard 1b (step S123). As a result, the transmitting / receiving unit 11b of the electronic whiteboard 1b receives the completion notification transmitted from the communication management system 5.

[0135] Next, the transmitting / receiving unit 11b of the electronic whiteboard 1b transmits request information indicating a request for downloading image data of the document image to the URL related to the image storage device 7 received in step S123 (step S124). As a result, the transmitting / receiving unit 71 of the image storage device 7 receives the request information transmitted from the electronic whiteboard 1b.

[0136] Next, in the image storage device 7, the storage / read processing unit 79 reads out image data of the requested document image from the storage unit 7000 based on the URL (step S125). Then, the transmission / reception unit 71 transmits the image data of the requested document image to the requesting electronic whiteboard 1b (step S126). As a result, the transmission / reception unit 11b of the electronic whiteboard 1b downloads (receives) the image data transmitted from the image storage device 7.

[0137] After the image data is downloaded, the display control unit 14b of the electronic whiteboard 1b displays the downloaded image on the display 180 of the electronic whiteboard 1b (step S127).

[0138] Next, on the electronic whiteboard 1b side, the display control unit 14b displays the same document image as the document image c1 shown in FIG. 19(C) on the screen shown in FIG. 19(D), and also hides the hourglass icon c2 that had been displayed until then (step S128).

[0139] Next, the transmitting / receiving unit 11b of the electronic whiteboard 1b transmits a download completion notification of the image (video) data to the communication management system 5 (step S129). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the download completion notification of the image (video) data transmitted from the electronic whiteboard 1b.

[0140] Then, the transmitting / receiving unit 51 transmits a download completion notification of the image (video) data to the electronic whiteboard 1a (step S130). As a result, the transmitting / receiving unit 11a of the electronic whiteboard 1a receives the download completion notification of the image (video) data transmitted from the communication management system 5.

[0141] <Flowchart of display switching process> In the communication system of this embodiment, a display switching process is performed for the sharing acceptance process of the document image (video) in step S101 described above among the processes according to the sequence diagrams illustrated in Figures 17 and 18. However, the process of step S101 in the series of processes illustrated in Figures 17 and 18 is only an example, and this embodiment can also be applied to a system that does not include both the electronic whiteboard 1a (starting terminal) and the electronic whiteboard 1b (destination terminal) and only one of the electronic whiteboards exists alone. Therefore, the display switching process performed on the electronic whiteboard 1a will be described in detail below.

[0142] Fig. 20 is a flowchart showing an example of a display switching process according to the present embodiment. As shown in Fig. 20, the reception unit 12a of the electronic whiteboard 1a waits for input from the user A1 to various selection buttons displayed on the display 180 (step S101-1). Note that the input from the user A1 can be made with the electronic pen 190, a hand H, or the like.

[0143] When the reception unit 12a receives an input to any of the selection buttons by the user A1, the judgment unit 15a judges whether the display content on the display 180 is a PC screen or an imported screen (capture screen, hereinafter referred to as a "capture screen") (step S101-2). The PC screen judged in this step S102-2 refers to a screen in which, for example, the "PC" button bc1 is operated and highlighted, or the outer frame of the "PC" button bc1 is displayed in a thicker color, as shown in Fig. 21(A) and Fig. 21(C). On the other hand, the capture screen refers to a screen in which, for example, the "Camera" button ba1 is operated and highlighted, or the outer frame of the "Camera" button ba1 is displayed in a thicker color, among the screen examples shown in Fig. 21.

[0144] If the display content is a PC screen (step S101-2; PC screen), the image processor 17a captures the PC screen (step S101-3). A specific example of the capture of the PC screen will be described in detail with reference to the flow chart shown in FIG.

[0145] Next, the image processor 17a creates display data for the captured screen from the image constituting the captured PC screen, to be written onto that image (step S101-4).

[0146] Thereafter, the display control unit 14a displays the display data for the captured screen created in step S101-4 on the display 180, and then exits this flow (step S101-5).

[0147] On the other hand, if the display content is an image related to a capture screen (step S101-2; captured screen), the display control unit 14a displays handwritten drawing data (stroke data) as a pen drawing on the display 180, and exits this flow (step S101-6).

[0148] The processing of the image processing unit 17a in the above-mentioned steps S101-3 and S101-4 is realized by a plurality of image layer structures provided in the electronic whiteboard 1a, for example, as shown in FIG. 3. That is, the image processing unit 17a copies the image of the pie chart (document image c1) displayed in the layer of the document image (L3) in FIG. 3(A) to the layer of the background image (L4), and erases the pie chart displayed in the layer of the (L3) (making the layer of the document image (L3) a transparent screen like the pie chart displayed with a dotted line). That is, the image of the pie chart displayed in the layer of the document image (L3) is moved to the layer of the background image (L4) (see FIG. 3(B)). As a result, the display control unit 14a displays (makes) the screen of the layer of the background image (L4) in FIG. 3(B) on the display 180, so that even if the image (video) displayed in the layer of the document image (L3) changes, the user A1 can see the screen in the screen-captured state.

[0149] <Flowchart of display switching process and mode switching process> Fig. 22 is a flowchart showing an example of a display switching process and a mode switching process according to the present embodiment. In Fig. 22, a process of determining whether the pen mode or the marker mode is set is added to the determination result of the PC screen or the captured screen described in Fig. 20. That is, in the present embodiment, the electronic whiteboards 1a and 1b are electronic whiteboards that are provided with a pen mode and a marker mode that can be set by the user A1.

[0150] The pen mode is a mode in which the user A1 writes on the display 180 using a pen 190 or the like. On the other hand, the marker mode is a mode in which a marker (also called a pointer) is displayed on the display 180 of the electronic whiteboard 1 by drawing a line or the like according to the movement of the user A1's hand H or the like. This marker is an image that is traced by the user A1's hand H or drawn with a dedicated touch pen or the like, and then erased from the display 180 in a few seconds (see FIG. 21(D)). In other words, the marker mode is a mode in which a drawing is made on the display 180 as if it were drawn on a piece of paper with a marker pen. The specifications for drawing in the marker mode are determined by whether or not it can be used with either the user A1's hand H, a dedicated touch pen (electronic pen 190), or both, depending on the model of the electronic whiteboard.

[0151] First, the processes of steps S101-1 and S101-2 in FIG. 22 are similar to the processes of steps S101-1 and S101-2 in FIG. 20, and therefore description thereof will be omitted.

[0152] Next, if the display content is a PC screen (step S101-2; PC screen), the determination unit 15a further determines whether the set mode is either the pen mode or the marker mode (step S101-11).

[0153] If the set mode is the pen mode (step S101-11; pen mode), the image processor 17a captures the PC screen (step S101-12). The process of this step S101-12 is similar to the process of step S101-3 described above.

[0154] Next, the image processing unit 17a creates display data for the captured screen to be written on the image from the image constituting the captured PC screen (creates a captured image) (step S101-13). The process of this step S101-13 is the same as the process of step S101-4 described above. In steps S101-12 and S101-13, the image processing unit 17a processes, for example, the images of the stroke image (L2) layer and the document image (L3) layer among the layers shown in FIG. 3(B) to be copied to the background image (L4) layer. In this embodiment, the following process is performed especially for the stroke image (L2) layer. That is, the image processing unit 17a performs a screen capture process when the user finishes drawing the curved solid line part of the curved arrow, and copies the curved solid line part of the curved arrow and the pie chart of the document image c1 to the background image (L4) layer.

[0155] Thereafter, the display control unit 14a displays the display data for the captured screen created in step S101-4 on the display 180, and then exits this flow (step S101-14). That is, the display content is switched to the content of the captured screen, and then the flow exits. The process of this step S101-14 is the same as the process of step S101-5 described above.

[0156] On the other hand, if the set mode is the marker mode (step S101-11; marker mode), the display control unit 14a displays the marker drawing data on the display 180 and exits this flow (step S101-15).

[0157] Here, even when the marker drawing process of step S101-15 (marker drawing process on the PC screen) is performed, image processing unit 17a can also perform screen capture of the input image input to the PC screen and execute the above-mentioned processes of steps S101-3, 4, and 5. That is, image processing unit 17a may perform the same screen capture process during image input in the marker mode as during pen input.

[0158] In step S101-2, if the display content is an image related to an imported screen (captured screen) (step S101-2; imported screen (captured screen)), the judgment unit 15a further judges whether the set mode is either the pen mode or the marker mode (step S101-16).

[0159] If the set mode is pen mode (step S101-16; pen mode), the image processing unit 17a performs drawing processing on the input image input by the electronic pen 190 or the like, and then the display control unit 14a displays the images of each image layer on the display 180 and exits this flow (step S101-17).

[0160] On the other hand, if the set mode is the marker mode (step S101-16; marker mode), the image processing unit 17a performs marker drawing processing, and then the display control unit 14a displays the images of each image layer on the display 180 and exits this flow (step S101-18).

[0161] <Flowchart of decision-making process when capturing a screen> Fig. 23 is a flowchart showing an example of a determination process during screen capture according to the present embodiment. In Fig. 23, a process for determining the input time and input length of an input image in pen mode is added to the display switching process and mode switching process described in Fig. 22. Note that the processes of steps S101-1, S101-2, and S101-11 in Fig. 23 are similar to the processes of steps S101-1, S101-2, and S101-11 in Fig. 22, and therefore will not be described.

[0162] If the judgment unit 15a judges in step S101-11 that the mode is the pen mode, the judgment unit 15a further judges whether the length of the line drawn by the pen or the time during which the pen is in contact with the surface of the display 180 (display device) is equal to or greater than a predetermined threshold (step S101-21). At this time, the judgment unit 15a works with the measurement unit 16a to measure the length of the line drawn by the pen and judges whether it is equal to or greater than a predetermined threshold. Furthermore, the judgment unit 15a works with the measurement unit 16a to measure the length of the line drawn by the pen and judges whether it is equal to or greater than a predetermined threshold. In step S101-21, if either the length or the time of the line drawn by the pen is equal to or greater than the respective predetermined threshold, the processing from step S101-12 onwards is performed. Note that the processing of steps S101-12, 13, and 14 in FIG. 23 is similar to the processing of steps S101-12, 13, and 14 in FIG. 22, and therefore corresponding detailed explanations will be omitted.

[0163] In this embodiment, even if the "PC" button bc1 is selected in step S101-2 of the flow chart in FIG. 23 and the PC screen is displayed on the display 180, that is, the "camera" button ba1 is not selected, if the judgment unit 15a judges in step S101-21 that the input of the input image is equal to or greater than a predetermined threshold (if a predetermined condition is satisfied), the image processing unit 17a captures the PC screen. As a result, even if the user A1 forgets to select the capture selection means (the "camera" button ba1 or the camera icon) and continues to input the input image to the screen, the PC screen is captured, so that even if the contents of the PC screen suddenly change (transition), it is possible to maintain the relationship with the input image input by the user A1. At this time, the display control unit 14a may display a thumbnail image of the capture screen from which the screen has been captured, for example, at the bottom of the display 180. Furthermore, the display control unit 14a may display an animation indicating that the screen has been captured in a time that does not affect the input operation of the input image by the user A1.

[0164] In this embodiment, the predetermined threshold value determined in step S101-21 is an example of the predetermined condition. The predetermined condition may be as follows.

[0165] ●The input time of the input image is longer than the specified time● The predetermined condition in this case is a case where the input time or distance related to the input of the input image is equal to or greater than a predetermined threshold. Specifically, as shown in FIG. 21(B), when the user A1 writes a curved arrow on the display 180 with the electronic pen 190 or the like, the reception unit 12a still receives the input of the input image even after the measurement unit 16a counts a predetermined time (for example, 0.1 seconds as a threshold) from when the reception unit 12a accepts (detects) that the reception unit 12a has started to draw the first curve. On the other hand, when the reception unit 12a stops accepting the input of the input image when the measurement unit 16a counts a predetermined time (for example, 0.04 seconds) from when the reception unit 12a accepts (detects) that the reception unit 12a has started to draw the curve, the judgment unit 15a judges that the predetermined condition is not satisfied. Such a judgment process functions as one of the so-called chattering prevention functions when an image is input to the display 180 of the electronic whiteboard 1.

[0166] ●The input distance (length) of the input image is greater than the specified distance (length)● The predetermined condition in this case is, as in the case of a predetermined time or more, when user A1 writes a curved arrow on display 180 with electronic pen 190 or the like, and even after reception unit 12a receives (detects) that the user A1 has started writing the first curve, a predetermined distance (length) (for example, 1 cm as a threshold) is measured by measurement unit 16a, reception unit 12a still receives input of an input image. In other words, in this case, an input image that is less than the threshold of 1 cm does not meet the predetermined condition for performing a screen capture.

[0167] As an application of the input distance (length), the measuring unit 16a may measure the area of ​​an image input by the user A1 using a change in coordinates provided on the display 180, and may import (take a screen capture) the input image when the area becomes equal to or larger than a predetermined area. As an application example of this case, it is mainly assumed that it will be used when the user A1 draws a figure such as a circle or a rectangle.

[0168] ●The interval between input images is longer than the specified time● The predetermined condition in this case is a case where the time interval between the input of an input image and the input of the next input image is equal to or greater than a predetermined threshold. Specifically, for example, when user A1 writes the character string "RICOH", a time interval (for example, 2 seconds as a threshold) equal to or greater than a predetermined threshold is provided between the end of writing the last character "H" and the input of the next input image. At this time, the measurement unit 16a measures the time that satisfies the above-mentioned condition using the processing of the CPU 101, the electronic pen controller 116, the sensor input to the contact sensor 114, and the sensor controller 115 shown in FIG. 4. In this case, in step S101-21 of FIG. 23, if the predetermined condition is not satisfied, the judgment unit 15a may repeat the judgment of step S101-21.

[0169] When input image input is first interrupted The predetermined condition in this case is when the reception of the input image to the display 180 (display means) is interrupted for the first time after the reception unit 12a (reception means) receives the input of the input image. Specifically, as shown in FIG. 21(C), for example, when the user A1 writes the character "R" in one stroke, the pen tip of the electronic pen 190 is removed from the screen of the display 180 after finishing writing the character "R" (when the input of the input image is interrupted). At this time, the measurement unit 16a uses the processing of the CPU 101, the electronic pen controller 116, the sensor input to the contact sensor 114, and the sensor controller 115 shown in FIG. 4 to determine the presence or absence of the contact of the electronic pen 190 or the like with the display 180, which is the above-mentioned condition. Furthermore, at this time, the display control unit 14a may change the highlight display of the selection button from the "PC" button bc1 to the "Pen" button bp1 as the screen after the screen capture processing. As shown in FIG. 21(C), in the screen capture process, the image processing unit 17a screen-captures an image including a pie chart data image (c1) including the stroke image of the character “R” input with the electronic pen 190.

[0170] The above-mentioned respective predetermined conditions may be appropriately incorporated into the program for the electronic whiteboard 1 in accordance with the usage pattern of the electronic whiteboard 1 and executed when the information communication system 8 is constructed.

[0171] If the determining unit 15a determines that the above-mentioned predetermined condition is satisfied, the image processing unit 17a captures the PC screen (step S101-3).

[0172] Next, the image processor 17a creates display data for the captured screen from the image constituting the captured PC screen, to be written onto that image (step S101-4).

[0173] Thereafter, the display control unit 14a displays the display data for the captured screen created in step S101-4 on the display 180, and then exits this flow (step S101-5).

[0174] 23, if the judgment unit 15a does not judge that the above-mentioned predetermined condition is satisfied (less than step S101-21), the image processing unit 17a erases the input image input to the display 180 (step S101-22). However, the image processing unit 17a may cooperate with the judgment unit 15a to return to the judgment process of step S101-21 instead of erasing the input image.

[0175] Furthermore, the processes of step S101-15 executed in the determination of the marker mode in the determination of step S101-11, and steps S101-17 and S101-18 executed in the determination of step S101-16 are similar to the processes of step S101-15 and step S101-17 and step S101-18 executed in the determination of step S101-16 described in Figure 22, and therefore corresponding detailed descriptions will be omitted.

[0176] <Screen clearing process> Here, a process of erasing screen data displayed on the display 180 of the electronic whiteboard 1a will be described. For example, as shown in FIG. 21(C), when a stroke image st1 indicated by the character "R" is drawn by the electronic pen 190 and the user A1 erases the character "R" with the eraser button or the like, the image processing unit 17a erases only the stroke image st1 indicated by "R". At this time, the image processing unit 17a leaves image data indicating the document image c1 indicated by a pie chart in FIG. 21(C) captured as a capture screen, various selection buttons such as the "camera" button ba1 or a camera icon, and an image of the "physical photoelectronic whiteboard" as an example of the background image d1. As a result, the display control unit 14a erases only the stroke image st1 indicated by "R" from the display 180.

[0177] The above-mentioned processing of the image processor 17a is realized by a structure of multiple image layers included in the electronic whiteboard 1a, for example, as shown in Fig. 3. That is, the image processor 17a erases only the curved arrow input to the layer of the shared stroke image (L2), and does not erase the images of the other image layers.

[0178] In this embodiment, the term "image input" is used as a general term for inputting characters and drawing figures and the like on the display 180 of the interactive whiteboard 1 using the electronic pen 190 or the like.

[0179] <About canceling the capture screen> In this embodiment, the capture screen executed in steps S101-3 to S101-5 may be released by the display control unit 14a when triggered by a change in the transmission data (image data) from the PC2, an operation on the pointing device 212 such as a mouse in the PC2, or the like. Furthermore, the capture screen may be released by an operation on the "PC" button bc1. On the other hand, when the capture screen is proceeded to based on the determination in step S101-2 of FIG. 20 (when the "camera" button ba1 is operated from the beginning, etc.), the display control unit 14a may control so as to maintain the capture screen even when a change in the transmission data (image data) from the PC2 has occurred, an operation on the pointing device 212 such as a mouse has occurred in the PC2, or the like.

[0180] In the present embodiment, the information communication system 8a including the PC 2a and the electronic whiteboard 1a has been described as an example, but the present invention can also be applied to an information communication system 8b including the PC 2b and the electronic whiteboard 1b.

[0181] [Main Effects of the Embodiments] As described above, the judgment unit 15a of the electronic whiteboard 1a judges whether the length or time of the line drawn by the pen is equal to or greater than the threshold (step S101-21), and if it is equal to or greater than the predetermined threshold, the image processing unit 17a captures the PC screen (step S101-3). Next, the image processing unit 17a creates display data for the captured screen to be written into the captured PC screen from the image constituting the captured PC screen (step S101-4). After that, the display control unit 14a displays the display data for the captured screen created in step S101-4 on the display 180 (step S101-5). This eliminates the need for the user A1 to operate (select) the "camera" button ba1 provided on the electronic whiteboard 1a in advance when attempting to capture a screen displayed on the electronic whiteboard 1a. In other words, even if the user A1 forgets to operate (select) the camera button ba1 and continues to input a predetermined input image with the electronic pen 190 or the like, the screen displayed on the display 180 is captured (screen captured) by satisfying a predetermined condition. Therefore, even if the image transmitted by the PC 2a changes after the screen capture, it is possible to maintain the relationship between the contents of the screen displayed on the display 180 and the contents of the input image being input by the user A1.

[0182] Furthermore, according to this embodiment, the judgment unit 15a of the electronic whiteboard 1a judges whether the length or time of the line drawn by the pen is equal to or greater than a threshold (step S101-21), and if it is equal to or greater than a predetermined threshold, the image processing unit 17a captures the PC screen (step S101-3). This is expected to have the effect of eliminating the need for the user to go back and redo the screen capture operation even if the user forgets to perform a screen capture operation (screen capture operation) on the PC screen and switches to another PC screen while inputting a stroke image on the PC screen.

[0183] [Supplementary description of the embodiment] In the above embodiment, an office device that is an electronic whiteboard has been described as an example of a communication terminal, but the present invention is not limited to this. Other examples of communication terminals include PCs, smartphones, smart watches, car navigation terminals, etc. Furthermore, the communication terminal includes medical equipment. In the case of medical equipment, the document image is an image of a patient. On the other hand, the communication terminal does not need to include the display 180 as a display means in the communication terminal itself. In other words, the communication terminal and the display 180 may have physically independent configurations (devices), and the display content of the display 180 may be controlled by remote operation by the communication terminal, etc.

[0184] In addition, in the above embodiment, a case where a video conference is held using a communication system is described, but the present invention is not limited to this, and the communication system may be used for meetings, general conversations between family members or friends, remote diagnosis, or one-way presentation of information.

[0185] In the above-described embodiment, the PC 2 may be of any type as long as it has a function of transmitting video (image) data, audio data, etc., which become material videos (images) to the electronic whiteboards 1a and 1b. For example, the PC 2 may be used as an external input device, examples of which include a smartphone and a tablet terminal.

[0186] Furthermore, each function of each of the above-mentioned embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU (Graphics Processing Unit), or a conventional circuit module designed to execute each of the above-mentioned functions. [Explanation of symbols]

[0187] 1a Electronic whiteboard (an example of a communication terminal) 1b Electronic whiteboard (an example of a communication terminal) 2a PC (an example of an information processing device) 2b PC (an example of an information processing device) 3. Relay Device 5. Communications Management System 7 Image storage device 8 Information and communication systems (examples of information and communication systems) 11a Transmitter / receiver unit (an example of a receiving means) 11b Transmitter / receiver unit (an example of a receiving means) 12a Reception unit (an example of a reception means) 12b Reception unit (an example of a reception means) 14a Display control unit (an example of a display control means) 14b Display control unit (an example of a display control means) 17a Image processing unit (an example of a capture means) 17b Image processing unit (an example of a capture means) ba1 camera button (an example of an import selection means) ba2 Camera button (an example of an import selection means) 180 Display (an example of a display means) [Prior art documents] [Patent documents]

[0188] [Patent Document 1] JP 2015-70543 A [Patent Document 2] JP 2019-133283 A

Claims

1. A communication terminal that displays on a display means a first screen on which an image represented by image data transmitted by an information processing device is displayed, a receiving means for receiving an input image drawn by inputting coordinates on the first screen; a capture means for capturing the first screen including the image represented by the image data and the input image as a captured image when the input of the input image accepted by the acceptance means satisfies a predetermined condition, and switching the content displayed on the display means from the first screen to a second screen on which the captured image is displayed; A communication terminal having the above configuration.

2. 2. The communication terminal according to claim 1, The display means; a display control means for, when erasing the input image while the image represented by the image data and the input image are being displayed on the display means, causing the image represented by the image data to remain and be displayed on the display means; A communication terminal having the above configuration.

3. the display means is provided with an image layer for displaying an image represented by the image data and an image layer for displaying the input image, The display control means When the input image is deleted, the contents of the image layer displaying the input image are deleted.

3. The communication terminal according to claim 2.

4. A communication terminal according to any one of claims 1 to 3, a capture selection means for selecting to capture the first screen including the image represented by the image data and the input image; having the capture means captures the first screen including the image represented by the image data and the input image when the input of the input image satisfies a predetermined condition even when the capture selection means is not selected. A communication terminal comprising:

5. A communication terminal according to any one of claims 1 to 4, The predetermined condition is that an input time or a distance related to the input of the input image is equal to or greater than a predetermined threshold. A communication terminal comprising:

6. A communication terminal according to any one of claims 1 to 4, The predetermined condition is that a time interval between the input of the input image and the input of the next input image is equal to or greater than a predetermined threshold value. A communication terminal comprising:

7. A communication terminal according to claim 2 or 3, the predetermined condition being a time when the acceptance of the input of the input image to the display means is first interrupted after the acceptance of the input of the input image to the display means by the acceptance means; A communication terminal comprising:

8. 8. The communication terminal according to claim 1, wherein the accepting unit accepts an input of the input image input by a predetermined tool used by a user or a part of the body of the user.

9. An information processing device; a communication terminal that displays on a display means a first screen on which an image represented by the image data transmitted by the information processing device is displayed; An information and communication system having: The communication terminal includes: a receiving means for receiving an input image drawn by inputting coordinates on the first screen; a capture means for capturing the first screen including the image represented by the image data and the input image as a captured image when the input of the input image accepted by the acceptance means satisfies a predetermined condition, and switching the content displayed on the display means from the first screen to a second screen on which the captured image is displayed; An information and communication system having the above configuration.

10. 1. An image processing method executed by a communication terminal that causes a display unit to display a first screen on which an image represented by image data transmitted by an information processing device is displayed, the method comprising: a receiving step of receiving an input image drawn by inputting coordinates on the first screen; a capture step of capturing the first screen including the image represented by the image data and the input image as a captured image when the input of the input image accepted in the acceptance step satisfies a predetermined condition, and switching the content to be displayed on the display means from the first screen to a second screen on which the captured image is displayed; An image processing method comprising:

11. A program for causing a computer to function as each of the means of the communication terminal according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Image processor, image processing system, image processing method and program

    JP2014035670A

  • Transmission terminal, transmission method, and program

    JP2015070543A

  • Information input system, information input device, information input method, and program

    JP2015106333A

  • Method for presenting association of content with stroke, computer program, and system

    JP2015165398A

  • Electronic blackboard, image processing method, image processing program, and image processing system

    JP2018067018A