Information processing apparatus, display method, display system, program

Abstract

To provide an information processing device which reports other information processing device recording image data or voice data.SOLUTION: An information processing device 10, which transmits image data or voice data to one or more other information processing devices 40 connected through a network, includes: reception means 11 which receives record notification, which indicates that the image data or the voice data transmitted from the information processing device is recorded by the other the information processing devices, together with the identification information of the other information processing devices; and record terminal notification means 18b which, based on the identification information received by the reception means, notifies that the other information processing devices which transmit the record notification is recording the image data or the voice data in a manner to associate with the other information processing devices.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a communication system, and a program. [Background technology]

[0002] There is known a transmission system for conducting a video conference or the like between a plurality of terminal devices via a network such as the Internet. In such a transmission system, image data and audio data collected during a video conference at one transmission terminal are transmitted to another transmission terminal. The other transmission terminal receives the image data and audio data, and displays the image on a display or the like and outputs the audio from a speaker, thereby enabling a video conference to be held between these transmission terminals.

[0003] In such a transmission system, the number of transmission terminals participating in a conference is not limited to two, and it is not rare that a video conference is held between multiple locations with three or more transmission terminals participating in the video conference.

[0004] As described above, each location can display video and output audio, but the transmission terminal at each location may have the function of recording video and audio (see, for example, Patent Document 1). This is to allow those who are unable to participate in a video conference to understand or confirm the contents of the conference later. Summary of the Invention [Problem to be solved by the invention]

[0005] However, since images and sounds transmitted in a video conference often contain highly confidential information, many participants do not like to be recorded or recorded. In addition, there are also participants who do not like their portraits being recorded in the images. In this regard, the communication system disclosed in Patent Document 1 is a technology in which each base records communication information in the same recording device, and there is a problem that when an arbitrary base is recording or recording, it is difficult for participants at other bases to identify the base that is recording or recording.

[0006] For example, if a video conference is being held at locations A to C, and location C is recording images or audio, the participants at location A will not know whether location B or C is recording, and the participants at location B will not know whether location A or C is recording. This makes it difficult to determine which location should be asked to stop recording, and whether highly confidential information should be disclosed.

[0007] SUMMARY OF THE PRESENT DISCLOSURE In view of the above-mentioned problems, an object of the present invention is to provide an information processing device that notifies another information processing device that is recording image data or audio data. [Means for solving the problem]

[0008] The present invention is an information processing device that transmits at least one of image data or audio data to one or more other information processing devices connected via a network, and includes a receiving means that receives a recording notification indicating that the other information processing device will record the image data or audio data transmitted by the information processing device along with identification information of the other information processing device, and a recording terminal notification means that notifies the other information processing device that the other information processing device that transmitted the recording notification is recording image data or audio data, in association with the other information processing device, based on the identification information received by the receiving means. Effect of the Invention

[0009] It is possible to provide an information processing device that notifies another information processing device that is recording image data or audio data. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of an outline of an operation of a transmission system. [Diagram 2] 1 is a schematic diagram of a transmission system according to an embodiment of the present invention; [Diagram 3] 1 is a conceptual diagram showing the state of transmission and reception of image data, audio data, and various management information in a transmission system. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a transmission terminal. [Diagram 5] 2 is a hardware configuration diagram of a transmission management device, a UI providing device, a relay device, or a program providing device. [Figure 6] FIG. 2 is an example of a functional block diagram of each terminal included in the transmission system. [Figure 7] FIG. 2 is an example of a functional block diagram of an information processing terminal. [Figure 8] An example is shown in which a transmitting terminal goes into a ready state. [Figure 9] 11 is a sequence diagram illustrating an example of a conference participation operation until a transmission terminal enters a participating state in a video conference. [Figure 10] 11 is a sequence diagram showing an example of an operation when the information processing terminal starts and ends recording; FIG. [Figure 11] FIG. 9 is a diagram showing an example of a destination list screen displayed in step S32 of FIG. 8. [Figure 12] 13 is a diagram illustrating an example of a confirmation dialogue screen displayed when a user selects a destination of a recording location mark. [Figure 13] FIG. 13 is a diagram showing an example of a screen displayed on a display in a full-screen display mode. [Figure 14] FIG. 13 is a diagram showing an example of an auxiliary area in a simplified display mode. [Figure 15] 13A and 13B are diagrams showing an example of how a different location recording mark and label are displayed when the auxiliary area is not displayed. [Figure 16] This is an example of a full-screen display screen in the case where none of the participating locations are recording. [Figure 17] 11 is a flowchart illustrating an example of a procedure in which a transmission terminal during a conference controls display of recording notification information. [Figure 18] 11 is an example of a sequence diagram illustrating an operation procedure of the transmission system 1 when the transmission management device detects that a session has been disconnected. [Figure 19] 11 is a flowchart illustrating a process performed when a transmission terminal receives terminal state change information; FIG. [Figure 20] FIG. 13 is a diagram showing an example of an information display screen displayed on a display in an information display mode. [Figure 21] 11 is a sequence diagram illustrating an example of a terminal information display operation for displaying terminal information in a terminal information display area when a transmission terminal is switched to an information display mode. FIG. [Figure 22] FIG. 1 is a diagram illustrating an example of an electronic whiteboard having a function of a transmission terminal. [Figure 23] FIG. 1 is a diagram for explaining an example of an outline of the operation of an electronic whiteboard; [Figure 24] 13 is an example of a diagram showing a display of an electronic whiteboard on which an electronic whiteboard UI screen generated by an electronic whiteboard application is displayed. [Diagram 25] FIG. 13 is a diagram illustrating an example of an electronic whiteboard display on which a video conferencing UI screen generated by a video conferencing application is displayed. [Figure 26] FIG. 2 is a hardware configuration diagram of the electronic whiteboard. [Figure 27] FIG. 2 is an example of a functional block diagram of an electronic whiteboard. [Figure 28] FIG. 2 is a diagram illustrating an example of the configuration of each image layer. [Figure 29] FIG. 2 is a diagram conceptually illustrating page data. [Diagram 30] FIG. 4 is a diagram conceptually illustrating stroke sequence data. [Diagram 31] FIG. 2 is a diagram conceptually showing coordinate array data. [Diagram 32] FIG. 2 is a diagram conceptually illustrating media data. [Diagram 33] 11 is a sequence diagram illustrating an example of an operation procedure when a video conferencing application running on an electronic whiteboard receives a recording start notification. [Diagram 34] 13 is an example of an interactive whiteboard UI screen that displays a message that another location is recording. [Diagram 35] 11 is a sequence diagram illustrating an example of a procedure in which an interface unit changes the display order of an electronic whiteboard UI screen and a video conference UI screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. EXAMPLES

[0012] 1 is an example of a diagram for explaining an outline of the operation of the transmission system 1 of the present embodiment. The transmission terminal 10aa is holding a video conference with the transmission terminal 10ba and the information processing terminal 40bd. The transmission terminal 10aa synthesizes images from the transmission terminal 10ba and the information processing terminal 40bd and displays them on the display 120 (such a screen is called a full-screen display screen 501). The transmission terminal 10ba and the information processing terminal 40bd also display the same image.

[0013] (1) When the information processing terminal 40bd of the present embodiment starts recording an image during a video conference, it transmits a recording start notification (an example of a recording notification) to the transmission terminals 10aa and 10ba. For convenience of explanation, the following description focuses on the transmission terminal 10aa.

[0014] (2) The transmission terminal 10aa holds the communication ID (04bd) of the information processing terminal 40bd contained in the recording start notification.

[0015] (3) When the user of the transmission terminal 10aa displays the information display screen 601 to check the location where the recording is being performed, the location where the recording is being performed is highlighted using the stored communication ID. In Fig. 1, the terminal name and communication ID are displayed, and a recording location mark 71 is displayed at the beginning of the terminal name of the information processing terminal 40bd that is performing the recording. The user of the transmission terminal 10aa can determine which of the multiple locations the recording is being performed at by using the recording location mark 71.

[0016] In this manner, in the transmission system 1 of the present embodiment, when a conference is being held at another base, participants at the other base can grasp, confirm, or identify in real time which base is recording.

[0017] <Terminology> The identification information of an information processing device is information for other information processing devices to identify the information processing device. Specifically, the information processing device may be identified by a communication ID. In addition, the information processing device may be identified by a terminal name, a MAC address, an email address, a user name, a user ID, or the like.

[0018] The information processing device is a device capable of transmitting and receiving at least one of image data and audio data. In this embodiment, the information processing device will be described using the terminology of a transmission terminal 10 or an information processing terminal 40.

[0019] Notifying another information processing device that recording is in progress in association with the other information processing device means outputting information that can identify the information processing device, such as the terminal name and communication ID of the information processing device, and that at least one of image data and audio data is being recorded. Output includes notification by sound in addition to display. In this embodiment, for example, a recording location mark 71 is displayed in association with the terminal name and communication ID.

[0020] Moreover, the recording of image data or audio data refers to a process expressed as preservation, retention, storage, or memory. Recorded image data or audio data can be played back. Furthermore, recording may also be referred to as video recording or audio recording.

[0021] The information on the other information processing devices refers to information that enables a participant to know the other participants who are participating in the conference, such as at least one of the terminal names and the communication IDs.

[0022] The term "in communication" refers to a state in which two or more transmission terminals 10 and the information processing terminal 40 are transmitting or receiving image data or audio data. Specifically, the term "in communication" refers to a state in which a video conference is being held or a state in which a participant is participating in a video conference.

[0023] A recording notification refers to a notification that recording has started or is currently being recorded, and a recording end notification refers to a notification that recording has ended.

[0024] <Overall configuration of the embodiment> FIG. 2 is a schematic diagram of a transmission system 1 according to this embodiment. FIG. 3 is a conceptual diagram showing the state of transmission and reception of image data, audio data, and various management information in the transmission system 1. The transmission system 1 includes a data providing system that transmits content data in one direction from one transmission terminal to multiple transmission terminals via a transmission management device 50. It also includes a communication system that transmits information, emotions, etc. between multiple transmission terminals via the transmission management device 50. The communication system is a system for mutually transmitting information, emotions, etc. between multiple communication terminals (corresponding to "transmission terminals") via a communication management device (corresponding to "transmission management device 50"). Examples include a video conference system and a video telephone system.

[0025] In this embodiment, a transmission system 1, a transmission management device 50, and a transmission terminal 10 will be described assuming a video conference system as an example of a communication system, a video conference management device as an example of a communication management device, and a video conference terminal as an example of a communication terminal. That is, the transmission terminal and the transmission management device 50 of the present invention are applied not only to a video conference system, but also to a communication system or a data transmission system.

[0026] 2, the transmission system 1 includes a plurality of transmission terminals 10aa, 10ab, ..., displays 120aa, 120ab, ... for each of the transmission terminals 10aa, 10ab, ..., a UI providing device 20, information processing terminals 40bd, be, a relay device 30, a transmission management device 50, and a program providing device 90. The transmission terminals 10aa, 10ab, ... are configured to transmit content data, for example, by sending and receiving image data and audio data.

[0027] The UI providing device 20 is a device that transmits layout information (to be described later) for mainly displaying the number of locations to the transmission terminals 10aa, 10ab, .... The UI providing device 20 may also be the transmission management device 50 or the relay device 30, for example.

[0028] The information processing terminal 40 is a device in which the functions of the transmission terminal 10 are realized by software. Specifically, the information processing terminal 40 is, for example, a personal computer (PC), a tablet terminal, a smartphone, a mobile phone, a personal digital assistant (PDA), a game machine, a car navigation device, or an electronic book terminal.

[0029] In the following description, any one of the multiple transmission terminals 10aa, 10ab, ... is represented as a "transmission terminal 10", and any one of the multiple displays 120aa, 120ab, ... is represented as a "display 120". Any one of the multiple information processing terminals 40bd, be is represented as an "information processing terminal 40". The transmission terminal 10 that requests the start of a video conference is represented as a "requesting terminal", and the transmission terminal 10 that is the destination of the request is represented as a "destination terminal".

[0030] In addition, the transmission terminal 10 participating in the video conference or the location where the transmission terminal 10 is assumed to be installed is referred to as a "base". Alternatively, it may be referred to as the other user who uses the transmission terminal 10. If the transmission terminal 10 participating in the video conference is identified, the user can estimate the location where the transmission terminal 10 is assumed to be installed and the other user. The user can estimate the location where the other party or the transmission terminal 10 is assumed to be installed by the terminal name described later. In addition, it is not necessary to identify the other party's location, and the location where the other party is located is the base. In addition, it is not necessary to identify the other party as an individual, and it is sufficient to know attributes such as the company name. For example, in corporate use, if the other party's transmission terminal 10 is identified, the user can estimate the other party and the other party's location. The number of bases is the same as the number of transmission terminals 10 participating in the video conference, but multiple transmission terminals 10 may exist in a space that can be considered as one (for example, one conference room).

[0031] As shown in FIG. 3, in the transmission system 1, a management information session sei for transmitting and receiving various types of management information is established between the transmission terminal 10aa, which is the request source terminal, and the transmission terminal 10ab (the symbol is just an example), which is the destination terminal, via the transmission management device 50.

[0032] Furthermore, between the transmission terminals 10 where the management information session sei has been established, four sessions for transmitting and receiving each of four data items, high-resolution image data sed1, medium-resolution image data sed2, low-resolution image data sed3, and audio data sed4, are established via the relay device 30. In this embodiment, these four sessions are collectively referred to as the "image and audio data session sed."

[0033] 2, the relay device 30 relays content data between a plurality of transmission terminals 10. The transmission management device 50 centrally manages login authentication from the transmission terminal 10, management of the transmission status of the transmission terminal 10, management of the destination list, and the transmission status of the relay device 30. Note that the image represented by the image data may be a moving image or a still image, or may be both a moving image and a still image.

[0034] The routers 70a, 70b, 70c, 70d, 70ab, and 70cd select the optimal route for image data and audio data. In the following description, any one of the routers 70a, 70b, 70c, 70d, 70ab, and 70cd will be referred to as a "router 70."

[0035] The program providing device 90 is equipped with an HD (Hard Disk) 204 described later, in which a transmission terminal control program 1000a for realizing various functions in the transmission terminal 10 is stored, and the program providing device 90 is capable of transmitting the transmission terminal control program 1000a to the transmission terminal 10.

[0036] Furthermore, HD 204 of the program providing device 90 also stores a relay device program for causing the relay device 30 to realize various functions, and can transmit the relay device program to the relay device 30.

[0037] Furthermore, the HD 204 of the program providing device 90 also stores a transmission management program 5000 a for causing the transmission management device 50 to realize various functions, and the transmission management program 5000 a can be transmitted to the transmission management device 50 .

[0038] Furthermore, the HD 204 of the program providing device 90 also stores a program for the UI providing device for causing the UI providing device 20 to realize various functions, and can transmit the program for the UI providing device to the UI providing device 20 .

[0039] Furthermore, the HD 204 of the program providing device 90 also stores a program for an information processing terminal 1000 c for implementing various functions in the information processing terminal 40 , and is capable of transmitting the program for an information processing terminal 1000 c to the information processing terminal 40 .

[0040] The transmission terminals 10aa, 10ab, 10ac, ... and the router 70a are connected to each other via a LAN 2a so as to be able to transmit data. The transmission terminals 10ba, 10bb, 10bc, ... and the router 70b are connected to each other via a LAN 2b so as to be able to transmit data.

[0041] Moreover, LAN2a and LAN2b are connected to each other so as to enable communication by a dedicated line 2ab including a router 70ab, and are provided within a predetermined area A. For example, area A is Japan, LAN2a is provided within an office in Tokyo, and LAN2b is provided within an office in Osaka.

[0042] On the other hand, the transmission terminals 10ca, 10cb, 10cc, . . . and the router 70c are connected to each other via a LAN 2c so that they can transmit data. The transmission terminals 10da, 10db, 10dc, . . . and the router 70d are connected to each other via a LAN 2d so that they can transmit data.

[0043] Furthermore, LAN2c and LAN2d are connected to each other so that communication can be performed by a dedicated line 2cd including a router 70cd, and are provided in a predetermined area B. For example, area B is the United States of America, with LAN2c provided in an office in New York and LAN2d provided in an office in Washington, D.C. Areas A and B are connected to each other so that communication can be performed by routers 70ab and 70cd, respectively, via the Internet 2i.

[0044] The relay device 30, the transmission management device 50, the UI providing device 20, the information processing terminal 40, and the program providing device 90 are connected to each transmission terminal 10 via the Internet 2i in a manner enabling communication therebetween. Here, the relay device 30, the transmission management device 50, the UI providing device 20, the information processing terminal 40, and the program providing device 90 may be installed in region A or region B, or may be installed in other regions.

[0045] In this embodiment, the transmission network 2 of this embodiment is configured by the LAN 2a, the LAN 2b, the dedicated line 2ab, the Internet 2i, the dedicated line 2cd, the LAN 2c, and the LAN 2d. The transmission network 2 may include a portion where not only wired transmission but also wireless transmission such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark) is performed.

[0046] 2, four sets of numbers shown under each transmission terminal 10, relay device 30, transmission management device 50, each router 70, UI providing device 20, information processing terminal 40, and program providing device 90 simply indicate general IP addresses in IPv4. For example, the IP address of the transmission terminal 10aa is "1.2.1.3". Although IPv6 may be used instead of IPv4, in order to simplify the explanation, IPv4 will be used in the explanation.

[0047] Each transmission terminal 10 may be used not only for video conferences among a plurality of offices or between different rooms in the same office, but also for video conferences in the same room, or for video conferences between indoors and outdoors or between outdoors and outdoors. When each transmission terminal 10 is used outdoors, wireless transmission is performed using a mobile phone transmission network or the like.

[0048] <Hardware configuration of transmission terminal> FIG. 4 is an example of a hardware configuration diagram of a transmission terminal. As shown in FIG. 4, the transmission terminal 10 has a CPU (Central Processing Unit) 101 that controls the overall operation of the transmission terminal 10, and a ROM (Read Only Memory) 102 that stores programs used to drive the CPU 101, such as an IPL (Initial Program Loader). The transmission terminal 10 also has a RAM (Random Access Memory) 103 that is used as a work area for the CPU 101, and a flash memory 104 that stores a transmission terminal control program 1000a, various data such as image data and audio data. The transmission terminal 10 also has an SSD (Solid State Drive) 105 that controls reading or writing various data from the flash memory 104 under the control of the CPU 101. The transmission terminal 10 also has a media drive 107 that controls reading or writing (storing) data from a recording medium 106 such as a flash memory, and an operation button 108 that is operated when selecting a destination of the transmission terminal 10, etc. The transmission terminal 10 also includes a power switch 109 for switching the power of the transmission terminal 10 on and off, and a network I / F (Interface) 111 for transmitting data using the transmission network 2.

[0049] The transmission terminal 10 also has a built-in camera 112 that captures an image of a subject under the control of the CPU 101 to obtain image data, an image sensor I / F 113 that controls the driving of the camera 112, and a microphone 114 as a built-in sound collecting device that inputs sound. The transmission terminal 10 also has a built-in speaker 115 that outputs sound, and a sound input / output I / F 116 that processes input and output of sound signals between the microphone 114 and the speaker 115 under the control of the CPU 101. The transmission terminal 10 also has a display I / F 117 that transmits image data to an external display 120 under the control of the CPU 101. The transmission terminal 10 also has an external device connection I / F 118 for connecting various external devices, and a bus line 110 such as an address bus and a data bus for electrically connecting the above components.

[0050] The display 120 constitutes a display device constituted by a liquid crystal or an organic electroluminescence (EL) displaying an image of a subject, operation icons, etc. The display 120 is connected to the display I / F 117 by a cable 120c. The cable 120c may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (registered trademark) (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signals.

[0051] The camera 112 includes a lens and a solid-state image sensor that converts light into electric charges to digitize an image of a subject. As the solid-state image sensor, a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) is used.

[0052] The external device connection I / F 118 allows external devices such as an external camera, an external microphone, and an external speaker to be connected via a USB (Universal Serial Bus) cable or the like.

[0053] When an external camera is connected to the external device connection I / F 118, the CPU 101 drives the external camera in priority over the built-in camera 112. When an external microphone or an external speaker is connected to the external device connection I / F 118, the CPU 101 drives the external microphone or the external speaker in priority over the built-in microphone 114 or the built-in speaker 115, respectively.

[0054] The recording medium 106 is configured to be detachable from the transmission terminal 10. The transmission terminal 10 may use an EEPROM (Electrically Erasable and Programmable ROM) or the like instead of the flash memory 104, as long as the EEPROM is a non-volatile memory that reads or writes data under the control of the CPU 101.

[0055] The transmission terminal control program 1000a may be distributed by being recorded in a computer-readable recording medium such as the recording medium 106 as a file in an installable or executable format. The transmission terminal control program 1000a may be stored in the ROM 102 instead of the flash memory 104.

[0056] The information processing terminal 40 has the same hardware configuration as the transmission terminal 10, or even if there is a difference, it is assumed that there is no problem at least in the description of this embodiment. However, the SSD 105 records an information processing terminal program 1000c for controlling the information processing terminal 40.

[0057] <<Hardware configuration diagram of the transmission management device 50, UI providing device, relay device, and program providing device>> FIG. 5 is a hardware configuration diagram of the transmission management device 50, the UI providing device 20, the relay device, or the program providing device. As shown in FIG. 5, the transmission management device 50 has a CPU 201 that controls the operation of the entire transmission management device 50, a ROM 202 that stores a program used to drive the CPU 201, such as an IPL, and a RAM 203 that is used as a work area for the CPU 201. The transmission management device 50 also has an HD 204 that stores various data such as a transmission management program, and an HDD (Hard Disk Drive) 205 that controls reading or writing of various data from the HD 204 under the control of the CPU 201. The transmission management device 50 also has a media drive 207 that controls reading or writing (storing) of data from a recording medium 206 such as a flash memory, and a display 208 that displays various information such as a cursor, a menu, a window, characters, or an image. The transmission management device 50 also has a network I / O 2809 for transmitting data using the transmission network 2, and a keyboard 211 with a plurality of keys for inputting characters, numbers, various instructions, and the like. The computer also has a mouse 212 for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. It also has an optical drive 214 for controlling reading and writing of various data from and to an optical storage medium (Compact Disc Read Only Memory) 213, which is an example of a removable storage medium. It also has bus lines 210 such as an address bus and a data bus for electrically connecting the above components.

[0058] The transmission management program may be distributed by being recorded in a computer-readable recording medium such as the recording medium 206 or the optical storage medium 213 as a file in an installable or executable format. The transmission management program may be stored in the ROM 202 instead of the HD 204.

[0059] Moreover, the relay device 30 has the same hardware configuration as the transmission management device 50, and therefore the description thereof will be omitted. However, the HD 204 stores a relay device program for controlling the relay device 30.

[0060] In this case, the program for the relay device may be distributed by being recorded in a computer-readable recording medium such as the recording medium 206 or the optical storage medium 213 as a file in an installable or executable format. The program for the relay device may be stored in the ROM 202 instead of the HD 204.

[0061] Moreover, the program providing device 90 has the same hardware configuration as the transmission management device 50, and therefore the description thereof will be omitted. However, the HD 204 stores a program for the program providing device for controlling the program providing device 90.

[0062] In this case as well, the program for the program providing device may be distributed by being recorded in a computer-readable recording medium such as the recording medium 206 or the optical storage medium 213 as a file in an installable or executable format. Also, the program for the program providing device may be stored in the ROM 202 instead of the HD 204.

[0063] Moreover, the UI providing device 20 has a hardware configuration similar to that of the transmission management device 50, and therefore the description thereof will be omitted. However, the HD 204 records a UI providing device program for controlling the UI providing device 20 and layout information to be described later.

[0064] In this case, the UI providing device program and the layout information may be distributed by being recorded in a computer-readable recording medium such as the recording medium 206 or the optical storage medium 213 as a file in an installable or executable format. Also, the UI providing device program and the layout information may be stored in the ROM 202 instead of the HD 204.

[0065] As the computer-readable recording medium, a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray disc, etc. may be used.

[0066] The illustrated hardware configuration of the transmission management device 50 etc. does not need to be housed in a single housing or provided as a single integrated device, but indicates hardware elements that are preferably provided in the transmission management device 50 etc. Also, in order to accommodate cloud computing, the physical configuration of the transmission management system 50 etc. of this embodiment does not need to be fixed, and may be configured such that hardware resources are dynamically connected and disconnected depending on the load.

[0067] <Functional configuration of transmission system 1> Fig. 6 is a functional block diagram of each terminal included in the transmission system 1. The transmission terminal 10, the relay device 30, the information processing terminal 40, the UI providing device 20, and the transmission management device 50 are connected so as to be able to transmit data via the transmission network 2. Note that the relay device 30 and the program providing device 90 shown in Fig. 2 will be described as necessary. The information processing terminal 40 will be described in Fig. 7.

[0068] <<Functional configuration of the transmission terminal>> The transmission terminal 10 has a transmission / reception unit 11, an operation input reception unit 12, a login request unit 13, an imaging unit 14, a sound input unit 15a, a sound output unit 15b, a destination list display unit 16, a display control unit 17, and a storage / readout processing unit 19. These units are functions realized by a CPU 101 that executes a transmission terminal control program 1000a expanded from the flash memory 104 in Fig. 4 onto the RAM 103. The transmission terminal 10 also has a storage unit 1000 that is configured by the RAM 103 and the flash memory 104. The storage unit 1000 stores the transmission terminal control program 1000a and a recording management table 1000b.

[0069] [Table 1] Table 1 shows an example of the recording in progress management table 1000b. The communication IDs of the transmitting terminal 10 and the information processing terminal 40 that are recording are registered in the recording in progress management table 1000b. That is, the transmitting terminal 10 and the information processing terminal 40 that have started recording transmit a recording start notification (including a communication ID) to the other transmitting terminals 10 and the information processing terminals 40, and the transmitting terminals 10 and the information processing terminals 40 that have received the recording start notification register their communication IDs in the recording in progress management table 1000b. When multiple transmitting terminals 10 and information processing terminals 40 are recording, the recording start notifications transmitted by the communication IDs of all the transmitting terminals 10 and the information processing terminals 40 that have transmitted the recording start notifications are held in the recording in progress management table 1000b.

[0070] For convenience of explanation, it is assumed that a communication ID starting with “01” represents the communication ID of the transmission terminal 10 and a communication ID starting with “04” represents the communication ID of the information processing terminal 40.

[0071] (Functional configuration of the transmission terminal) The transmission / reception unit 11 is composed of a CPU 101, a network I / F 111, etc., and is configured to transmit and receive various data and information with other transmission terminals 10, information processing terminals 40, relay devices 30, transmission management devices 50, or program providing devices 90 via the transmission network 2.

[0072] Before starting a video conference with a desired destination terminal, the transmitting / receiving unit 11 starts receiving each status information indicating the status of each transmitting terminal 10 as a candidate for a connection destination of the video conference from the transmission management device 50. The status information not only indicates the operating status of each transmitting terminal 10, whether it is online or offline, but also indicates detailed operating status such as whether it is online and capable of transmitting, whether it is online and transmitting, etc.

[0073] The status information indicates not only the operating status of each transmission terminal 10 but also various other states, such as the cable 120c being disconnected from the transmission terminal 10, outputting sound but not outputting images, not outputting sound (MUTE), etc. In the following description, in order to facilitate understanding of the present invention, a case will be described in which the status information indicates the operating status of each transmission terminal 10.

[0074] The operation input reception unit 12 is configured with a CPU 101, an operation button 108, a power switch 109, etc., and is configured to receive various inputs from a user. For example, when the user turns on the power switch 109, the operation input reception unit 12 receives the power ON command and turns on the power of the transmission terminal 10.

[0075] The login request unit 13 is configured with the CPU 101 etc. When the power is turned on, the login request unit 13 transmits login request information indicating a login request and the current IP address of the terminal from the transmitting / receiving unit 11 to the transmission management device 50 via the transmission network 2.

[0076] In addition, when the user turns the power switch 109 from an ON state to an OFF state, the transmitting / receiving unit 11 transmits state information to the transmission management device 50 to the effect that the power is to be turned OFF, and then the operation input receiving unit 12 completely turns OFF the power of the transmission terminal 10. This allows the transmission management device 50 to know that the transmission terminal 10 has been turned OFF from a power ON state.

[0077] The imaging unit 14 is made up of a CPU 101, a camera 112, an imaging element I / F 113, etc., and is configured to capture an image of a subject and output image data obtained by capturing the image to the transmitting / receiving unit 11.

[0078] The sound input unit 15a is composed of a CPU 101, a sound input / output I / F 116, etc., and is configured to input a sound signal representing a sound collected by a microphone 114 and convert the input sound signal into audio data.

[0079] The sound output unit 15b is composed of the CPU 101, a sound input / output I / F 116, etc., and converts audio data into a sound signal and outputs the converted sound signal to the speaker 115, thereby causing the speaker 115 to output sound.

[0080] The display control unit 17 is composed of the CPU 101, the display I / F 117, etc. The display control unit 17 combines image data with different resolutions received by the transmitting / receiving unit 11, and transmits the combined image data to the display 120, thereby causing the display 120 to display an image represented by the image data (full screen display mode). In addition, detailed information of other bases can be displayed (information display mode).

[0081] The display control unit 17 has an auxiliary area display unit 18a and a base list display unit 18b. When the transmission terminal 10 is in a full screen display mode described later, the auxiliary area display unit 18a displays the number of bases participating in the conference, recording notification information (recording notification information), and the like in the auxiliary area of ​​the full screen display screen 501. When the transmission terminal 10 is in an information display mode described later, the base list display unit 18b displays the names of the bases participating in the conference, communication IDs, and recording base marks 71, and the like on the information display screen 601 by referring to the recording management table 1000b.

[0082] The destination list display unit 16 displays a destination list screen on the display 120 based on the destination status information transmitted from the transmission management device 50. The destination status information includes an operating state of whether or not a recording conference is being held, and the destination list screen displays whether or not each of the destination transmission terminals 10 and information processing terminals 40 is in a conference.

[0083] The storage / read processing unit 19 is constituted by a CPU 101, an SSD 105, etc., and is configured to store various data in the storage unit 1000 and to read out various data stored in the storage unit 1000.

[0084] The storage unit 1000 stores a communication ID and a password for identifying the transmission terminal 10 as a communication destination (seen from another base). Furthermore, the storage unit 1000 is also used as a receiving buffer for image data and audio data received when a video conference is held with a destination terminal. Note that the communication ID in this embodiment indicates information such as language, characters, symbols, or various marks used to uniquely identify the transmission terminal 10. The communication ID may also be information that combines at least two of the language, characters, symbols, and various marks.

[0085] <Functional configuration of the transmission management device 50> The transmission management device 50 has a transmitting / receiving unit 51, a terminal authentication unit 52, a status management unit 53, a terminal extraction unit 54, a terminal status acquisition unit 55, a session management unit 56, and a storage / readout processing unit 59. Each of these units is a function realized by a CPU 201 that executes a transmission management program 5000a expanded onto the RAM 203 from the HD 204 shown in Fig. 5. The transmission management device 50 also has a storage unit 5000 configured by the HD 204. The database stored in the storage unit 5000 will be described below.

[0086] [Table 2] A terminal authentication management DB 5001 having a terminal authentication management table as shown in Table 2 is constructed in the storage unit 5000. In the terminal authentication management table, each password is associated with each communication ID of all transmission terminals 10 managed by the transmission management device 50. For example, the terminal authentication management table shown in Table 1 indicates that the communication ID of the transmission terminal 10aa is "01aa" and the password is "aaaa."

[0087] [Table 3] Moreover, the storage unit 5000 has constructed a terminal management DB 5002 having a terminal management table as shown in Table 3. In the terminal management table, the terminal name of each transmission terminal 10 or the information processing terminal 40, the operating status of each transmission terminal 10, and the IP address of the transmission terminal 10 are associated and managed for each communication ID of each transmission terminal 10. The terminal name is registered in advance in the transmission management device 50 in association with the communication ID, and the terminal name can be determined when the communication ID is known.

[0088] For example, the terminal management table shown in Table 3(a) indicates that for the transmission terminal 10aa with the communication ID "01aa", the terminal name is "AA terminal, Tokyo office, Japan", the operating status is "online (transmitting)", and the IP address of the transmission terminal 10aa is "1.2.1.3".

[0089] The terminal management table in Table 3(a) shows a state in which the transmission terminal 10aa with the communication ID 01aa, the transmission terminal 10ba with the communication ID 01ba, and the information processing terminal 40bd with the communication ID 04bd are participating in the conference (online (transmitting)). An earlier operating state is online (transmittable).

[0090] Moreover, Table 3(b) shows the terminal management table in the case where the information processing terminal 40 participating in the conference during the recording of the conference is participating in the conference. The operating status of the information processing terminal 40db with the communication ID "04db" is "recording conference". This operating status of "recording conference" indicates that the information processing terminal 40bd is recording. The operating status of the communication IDs "01aa" and "01ba" is also "recording conference". This is because the operating status of the transmission terminal 10 and the information processing terminal 40 participating in the same conference as the information processing terminal 40 is also changed to "recording conference" (even if the transmission terminal 10 and the information processing terminal 40 are not actually recording).

[0091] Here, the communication ID in the terminal management table is uniquely assigned when the transmission terminal 10 and the information processing terminal 40 are registered in the transmission system 1, and the terminal name is registered or changed by the administrator of the transmission management device 50. Note that the terminal name may be registered or changed in response to a request made from the transmission terminal 10 to the transmission management device 50. That is, the following description will be given assuming that the communication ID and the terminal name are registered in advance.

[0092] Additionally, the operating status "online (transmitting)" indicates that the transmission terminal 10 is logged in to the transmission system 1 and is not participating in a video conference. "Online (transmitting)" indicates that the transmission terminal 10 is logged in to the transmission system 1 and is participating in a video conference (meaning in a conference). The operating status "offline" indicates that the transmission terminal 10 is not logged in to the transmission system 1. A transmission terminal 10 that is participating in or in a video conference is a transmission terminal 10 whose operating status is "online (transmitting)".

[0093] [Table 4] Furthermore, the storage unit 5000 has constructed a destination list management DB 5003 having a destination list management table as shown in Table 4. In the destination list management table, the communication IDs of all destination terminals registered as candidates for the destination terminal are associated with the communication ID of a request source terminal requesting the start of a video conference and are managed.

[0094] For example, it is shown that the candidates of the destination terminals to which the transmission terminal 10aa with the communication ID "01aa" can request the start of a video conference are the transmission terminals 10ab to 10dc with the communication IDs "01ab" to "01dc" and the information processing terminals 40bd and be with the communication IDs "04bd" and "04be". The communication ID of the destination terminal is added when a permission notice (approval) is returned from the destination transmission terminal 10 or the information processing terminal 40 in response to a request made from the transmission terminal 10 or the information processing terminal 40 to the transmission management device 50. Also, the communication ID of the destination terminal can be deleted from the transmission terminal 10 or the information processing terminal 40 to the transmission management device 50.

[0095] [Table 5] The storage unit 5000 also has a session management DB 5004 configured by a session management table. In this session management table, for each session ID used to execute a session for selecting a relay device 30, the relay device ID of the relay device 30 used to relay image data and audio data, the communication ID of the request source terminal, the communication ID of the destination terminal, and the conference ID are associated and managed. The session ID is an ID for identifying a session, and transmitting terminals 10 participating in the same conference are not necessarily included in the same session. In contrast, the same conference ID is given to transmitting terminals 10 participating in the same conference.

[0096] (URI of UI providing device) Furthermore, the storage unit 5000 stores a uniform resource identifier (URI) 5005 of the UI providing device 20. The URI 5005 of the UI providing device 20 is set by the transmission management device 50 or an administrator of the UI providing device 20, or the like.

[0097] (Recording management table) The storage unit 5000 stores an in-recording management table 5006. In the in-recording management table 5006 held by the transmission management device 50, all communication IDs of the transmission terminal 10 and the information processing terminal 40 that are in the process of recording for each conference are registered.

[0098] (Functional configuration of the transmission management device 50) The transmission / reception unit 51 is constituted by a CPU 201, a network I / F 2809, etc., and transmits and receives various data and information to and from other transmission terminals 10, relay devices 30, and program providing devices 90 via the transmission network 2.

[0099] The terminal authentication unit 52 is configured by the CPU 101 etc., and searches the terminal authentication management DB 5001 using the communication ID and password included in the login request information received via the transmission / reception unit 51 as search keys. Then, it performs terminal authentication by determining whether the same communication ID and password are managed in the terminal authentication management DB 5001. Note that the authentication method is not limited to this, and a client certificate (authentication method using a public key and a private key) may also be used.

[0100] The status management unit 53 is configured by the CPU 101 etc., and manages the operation status of the request source terminal that has made the login request. That is, the operation status and the IP address of the request source terminal are stored and managed in the terminal management DB 5002 in association with the communication ID of the request source terminal.

[0101] Also, the user changes the state of the power switch 109 of the transmission terminal 10 from ON to OFF. The state management unit 53 changes the operation state of the transmission terminal 10 in the terminal management DB 5002 from online to offline based on the state information sent from the transmission terminal 10 to turn the power OFF. Also, the state management unit 53 periodically (or irregularly) performs communication to monitor the alive status of the transmission terminal 10 and the information processing terminal 40.

[0102] The terminal extraction unit 54 is configured by the CPU 101 etc., and searches the destination list management DB 5003 using the communication ID of the requesting terminal that made the login request as a search key, and extracts the communication IDs of candidate destination terminals for the requesting terminal.

[0103] In addition, the terminal extraction unit 54 searches the destination list management DB 5003 using the communication ID of the requesting terminal that has made the login request as a search key, and also extracts the communication IDs of other requesting terminals that have registered the communication ID of the requesting terminal as a candidate destination terminal.

[0104] The terminal status acquisition unit 55 is composed of the CPU 101 etc., and searches the terminal management DB 5002 using the communication ID of the candidate destination terminal extracted by the terminal extraction unit 54 as a search key, and reads out the operating status for each detected communication ID.

[0105] This allows the terminal status acquisition unit 55 to acquire the operating status of candidate destination terminals for the request source terminal that has made the login request. The terminal status acquisition unit 55 also searches the terminal management DB 5002 using the communication ID extracted by the terminal extraction unit 54 as a search key, and acquires the operating status of the request source terminal that has made the login request.

[0106] When a session is established between the transmitting terminal 10 and the transmitting terminal 10, between the transmitting terminal 10 and the information processing terminal 40, or between the information processing terminal 40 and the information processing terminal 40, the session management unit 56 manages the session in a session management table. When a different session is established for the transmitting terminal 10 or the information processing terminal 40 with which a session has already been established, the session management unit 56 assigns the same conference ID, so that the transmitting terminal 10 and the information processing terminal 40 participating in the same conference are linked by the conference ID.

[0107] The storage / readout processing unit 59 is constituted by the CPU 101, the HDD 205, etc., and is configured to store various data in the storage unit 5000 and to read out various data stored in the storage unit 5000.

[0108] <<Functional Configuration of UI Providing Device>> The UI providing device 20 includes a transmission / reception unit 81, a server unit 82, and a storage / reading processing unit 89. Each of these units is a function realized by a CPU 201 that executes a UI providing device program 2000a expanded from the HD 204 onto the RAM 203. Also, the transmission management device 50 includes a storage unit 2000 constituted by the HD 204. Layout information lot is stored in the storage unit 2000.

[0109] (Functional Configuration of UI Providing Device) The transmission / reception unit 81 is constituted by the CPU 201 and a network I / F 2809, etc., and is configured to transmit and receive various data and information to and from the transmission terminal 10 via the transmission network 2.

[0110] The server unit 82 is constituted by a CPU 101, etc., acquires a layout information request signal from the transmission terminal 10, and transmits the layout information lot read by the storage / reading processing unit 29 from the storage unit 2000 to the transmission terminal 10 via the transmission / reception unit 81.

[0111] The storage / reading processing unit 89 is constituted by a CPU 101, etc., and is configured to store various data in the storage unit 2000 or read out various data stored in the storage unit 2000. The storage unit 2000 is constituted by a ROM 202, a RAM 203, an HDD 205, etc., and mainly stores the UI providing device program 2000a and the layout information lot.

[0112] <<Functional Configuration of Information Processing Terminal>> 7 is an example of a functional block diagram of the information processing terminal 40. The functions of the information processing terminal 40 will be described mainly with respect to differences from the transmitting terminal 10. The functions of the information processing terminal 40 are similar to those of the transmitting terminal 10, but the information processing terminal 40 differs from the transmitting terminal 10 in that the information processing terminal 40 has an image recording unit 41. Each function of the information processing terminal 40 is realized by the CPU 101 that executes the information processing terminal program 1000c expanded on the RAM 103 from the flash memory 104 shown in FIG. 4. Furthermore, the information processing terminal 40 stores the information processing terminal program 1000c and recording data 1000d in the storage unit 1000.

[0113] The image recording unit 41 records the image data and audio data displayed on the display 120 as recorded data 1000d. The recorded data is data in which the full screen display screen 501 or the information display screen 601 displayed on the display 120 by the display control unit 17 is recorded. It is possible to save any screen displayed on the display 120, not limited to the full screen display screen 501 or the information display screen 601. It is also possible to record (record) audio data output from the speaker 115 by the sound output unit 15b. In this embodiment, it is assumed that audio data is also recorded by the image recording unit 41 recording. On the other hand, the image recording unit 41 is capable of recording only image data or only audio data.

[0114] In this embodiment, the information processing terminal 40 has a recording function, and the transmission terminal 10 does not have a recording function, but this is merely a distinction for the purpose of explanation, and it is also possible for the transmission terminal 10 to have a recording function.

[0115] Furthermore, the recorded data 1000d may be stored not in the storage unit 1000 but in a network attached storage (NAS) on a network.

[0116] <Processing or Operation of the Embodiment> Next, the operation of the transmission system 1 according to the present embodiment will be described. Fig. 8 is a sequence diagram showing a preparatory stage operation until the transmission terminal 10 is in a preparatory state to participate in a video conference. Fig. 8 shows an example in which the transmission terminal 10aa is in a preparatory state. In the preparatory stage operation, various information is transmitted and received in the management information session sei. The process in which the information processing terminal 40 participates in a video conference is also similar to that of the transmission terminal 10.

[0117] First, when a user of the transmission terminal 10aa turns on the power switch 109 of the transmission terminal 10aa, the operation input reception unit 12 receives the power-on command and turns on the power (step S21). Then, upon receiving the power-on command, the login request unit 13 transmits login request information representing a login request from the transmission / reception unit 11 of the transmission terminal 10aa to the transmission management device 50 via the transmission network 2 (step S22).

[0118] The login request information includes a communication ID and a password for identifying the transmission terminal 10aa that has made the login request. The communication ID and the password are read from the storage unit 1000 by the storage / read processing unit 19.

[0119] When the login request information transmitted from the transmission terminal 10aa is received by the transmission management device 50, the transmission management device 50 can know the IP address of the transmission terminal 10aa, which is the sender.

[0120] Next, the terminal authentication unit 52 of the transmission management device 50 searches the terminal authentication management DB 5001 of the storage unit 5000 using the communication ID and password included in the login request information received via the transmitting / receiving unit 51 as search keys. Then, the terminal authentication unit 52 performs terminal authentication by determining whether the communication ID and password identical to the search key are managed in the terminal authentication management DB 5001 (step S23).

[0121] Here, there may be a case where the terminal authentication unit 52 determines that the same communication ID and password as the search key are not managed, that is, that the login request is not from a transmission terminal 10 having a valid usage authority. In this case, the transmitting / receiving unit 51 transmits authentication result information indicating the authentication result obtained by the terminal authentication unit 52 to the transmission terminal 10aa that has made the login request via the transmission network 2, and the operation of the preparation stage is terminated.

[0122] On the other hand, there may be a case where the terminal authentication unit 52 determines that the same communication ID and password as the search key are managed, that is, that the login request is from a transmission terminal 10 with valid usage authority. In this case, the state management unit 53 stores the communication ID, operating state, and IP address of the transmission terminal 10aa in the terminal management DB 5002 in association with each other (step S24).

[0123] As a result, the terminal management table manages the communication ID "01aa", the operation state "online (transmission possible)", and the IP address "1.2.1.3" of the transmission terminal 10aa in association with each other.

[0124] Next, the transmitting / receiving unit 51 of the transmission management device 50 transmits authentication result information indicating the authentication result obtained by the terminal authentication unit 52 to the transmission terminal 10aa that has requested the login via the transmission network 2 (step S25). In addition, the transmission management device 50 transmits the URI 5005 of the UI providing device 20 to the transmission terminal 10 together with or before or after the authentication result information. This allows the transmission terminal 10 to access the UI providing device 20.

[0125] The URI 5005 of the UI providing device 20 indicates the storage location of the layout information lot in the transmission network 2. For example, it has a format such as "IP address (or domain name) + folder name + file name", but it is sufficient if the UI providing device 20 can be accessed.

[0126] Since each transmission terminal 10 does not hold the URI 5005, even if the UI providing device 20 is replaced or the URI is changed, the administrator of the transmission management device 50 only needs to change the URI stored in the transmission management device 50. In other words, since it is not necessary to change all the transmission terminals 10, the management load can be reduced.

[0127] Next, the terminal extraction unit 54 of the transmission management device 50 searches the destination list management DB 5003 using the communication ID “01aa” of the transmission terminal 10aa that has made the login request as a search key, and extracts communication IDs of candidates of destination terminals for the transmission terminal 10aa (step S26).

[0128] In this case, the communication IDs "01ab, 01ac, 01ba, 01bb, 01bc, 04bd, 04be, 01ca, 01cb, 01ca, 01da, 01db, and 01dc" are extracted as destination terminals for the communication ID "01aa" of the transmission terminal 10aa.

[0129] Next, the terminal status acquisition unit 55 of the transmission management device 50 searches the terminal management DB 5002 using the communication IDs "01ab, 01ac, 01ba, 01bb, 01bc, 04bd, 04be, 01ca, 01cb, 01ca, 01da, 01db, 01dc" of the destination terminal candidates extracted by the terminal extraction unit 54 as search keys. Then, the terminal status acquisition unit 55 acquires the respective operation statuses of the transmission terminal 10 and the information processing terminal 40 by reading out the operation statuses "offline", "online (transmittable)", "online (transmitting)", and "recording conference" for each communication ID extracted by the terminal extraction unit 54 (step S27).

[0130] Next, the transmitting / receiving unit 51 of the transmission management device 50 adds the communication IDs "01ab, 01ac, 01ba, 01bb, 01bc, 04bd, 04be, 01ca, 01cb, 01ca, 01da, 01db, 01dc" which are the search keys used in step S27, and the operation states of each transmission terminal 10 and the information processing terminal 40, "offline", "online (transmittable)", "online (transmitting)", and "recording conference in progress", to the destination state information, and then transmits the information to the transmission terminal 10aa via the transmission network 2 (step S28).

[0131] As a result, the transmitting terminal 10aa can grasp the current operating status of the transmitting terminals 10ab, 10ac, 10ba, 10bb, 10bc, 10ca, 10cb, 10ca, 10da, 10db, 10dc, and the information processing terminals 40bd and 40be, which are candidates for the destination terminals, namely "offline", "online (transmission possible)", "online (transmitting)", or "recording conference in progress".

[0132] Furthermore, the terminal extraction unit 54 of the transmission management device 50 searches the destination list management DB 5003 using the communication ID "01aa" of the transmission terminal 10aa that has made the login request as a search key. Then, the terminal extraction unit 54 extracts the communication IDs of other transmission terminals that have registered the communication ID "01aa" of the transmission terminal 10aa as a candidate for a destination terminal (step S29). According to the destination list management table of Table 4, "01ba", "01ca", and "04bd" are extracted.

[0133] Next, the status management unit 53 of the transmission management device 50 searches the terminal management DB 5002 using the communication ID “01aa” of the transmission terminal 10aa that has requested the login as a search key. Then, the status management unit 53 acquires the operation status “online (transmission possible)” of the transmission terminal 10aa that has requested the login (online (transmission possible) immediately after login) (step S30).

[0134] Next, the transmitting / receiving unit 51 of the transmission management device 50 identifies the transmission terminal 10ba and the information processing terminal 40bd whose operating status is "online" in the terminal management DB 5002 from among the transmission terminals 10ba, 10ca, and the information processing terminal 40db identified by the communication IDs "01ba," "01ca," and "04bd" extracted in step S29. This is because the destination status information cannot be received unless the terminal is online. Then, the transmitting / receiving unit 51 transmits destination status information including the communication ID "01aa" and operating status "online (transmission possible)" of the transmission terminal 10aa acquired in step S30 (steps S31-1, 31-2).

[0135] The destination list display unit 16 of the transmission terminal 10aa uses the destination status information received in step S28 to display a destination list screen on the display 120aa (S32). The destination list screen is shown in FIG.

[0136] When transmitting destination status information to the transmission terminal 10ba and the information processing terminal 40bd, the transmission / reception unit 51 refers to the IP addresses of the transmission terminals managed in the terminal management table based on the respective communication IDs “01ba” and “04bd”.

[0137] This makes it possible to communicate the communication ID "01aa" and the operating status "online (transmission possible)" of the transmission terminal 10aa that made the login request to the transmission terminal 10db and the information processing terminal 40bd, which can transmit to the transmission terminal 10aa that made the login request.

[0138] Furthermore, when step S28 of the preparation stage operation is completed, the display control unit 17 of the transmission terminal 10aa highlights the terminals participating in the video conference on the destination list screen displayed on the display 120. That is, the information of the transmission terminal 10ba and the information processing terminal 40bd whose operating status is "online (transmitting)" or "online (transmittable)" is highlighted on the display 120, so that the user can recognize the transmission terminals 10 participating in the video conference or the transmission terminals 10 that can participate.

[0139] Fig. 9 is a sequence diagram showing a conference participation operation until the transmission terminal 10 enters a participation state for participating in a video conference. Fig. 9 shows an example in which the transmission terminal 10aa holds a video conference with the information processing terminal 40bd. In the conference participation operation, various information is transmitted and received through the management information session sei.

[0140] First, when the user presses the operation button 108 to select, for example, the information processing terminal 40bd and thereby selects participation in a video conference, the operation input receiving unit 12 receives a request to participate in a video conference with the information processing terminal 40bd (step S41). Note that, when the transmission terminal 10aa is the information processing terminal 40, the request to participate in a video conference is received by pressing a hard key or a soft key displayed on a touch panel of the information processing terminal 40 instead of the operation button 108.

[0141] In response to this request, the transmitter / receiver unit 11 of the transmission terminal 10aa transmits participation request information to the transmission management device 50, which includes the communication ID "01aa" of the transmission terminal 10aa and the communication ID "04bd" of the information processing terminal 40bd and indicates an intention to participate in the video conference (step S42).

[0142] As a result, the transmitting / receiving unit 51 of the transmission management device 50 can receive the participation request information and confirm the IP address "1.2.1.3" of the transmission terminal 10aa which is the sender.

[0143] Next, the status management unit 53 of the transmission management device 50 uses the communication ID "01aa" of the transmission terminal 10aa included in the participation request information as a search key. Then, the status management unit 53 searches the terminal management DB 5002 and sets the operation status of the transmission terminal 10aa that has requested participation to "online (transmitting)" (step S43). As a result, the operation status in the terminal management table is updated as shown in Table 3(a).

[0144] Next, the transmitting / receiving unit 51 of the transmission management device 50 transmits start request information including the communication ID “01aa” of the transmission terminal 10aa to the information processing terminal 40bd via the transmission network 2 (step S44). This allows the information processing terminal 40bd to know which transmission terminal 10 has requested to participate in the video conference.

[0145] Next, the information processing terminal 40bd transmits reception completion information indicating that reception of the start request information is completed to the transmission management device 50 from the transmitting / receiving unit 11 via the transmission network 2 (step S45). The session management unit 56 assigns (numbers) a unique session ID to the session between the transmission terminal 10aa, which is the request source, and the information processing terminal 40bd, which is the destination terminal. Also, the session management unit 56 assigns a unique conference ID and registers the session ID, relay device ID, request source communication ID, destination communication ID, and conference ID in the session management table.

[0146] Next, the transmitter / receiver 51 of the transmission management device 50 transmits relay start request information representing a request to start relaying to the relay device 30 via the transmission network 2 (step S46). The relay start request information includes the IP addresses “1.2.1.3” and “1.3.1.3” of the transmission terminal 10aa and the information processing terminal 40bd, the session ID, and the conference ID.

[0147] As a result, the relay device 30 establishes a session for transmitting three types of image data, low resolution, medium resolution, and high resolution, and audio data between the transmission terminal 10aa and the information processing terminal 40bd (step S47). In this way, the transmission terminal 10aa can participate in a video conference with the information processing terminal 40bd.

[0148] 9, when a start request is sent to the transmission management device 50 for the transmission terminal 10 or the information processing terminal 40 that communicates in the same session, the session management unit 56 of the transmission management device 50 assigns the conference ID of the transmission terminal 10 and the information processing terminal 40 that have started the session. Therefore, even if the session ID changes, the same conference ID is assigned to the transmission terminal 10 or the information processing terminal 40 that participates in the same conference. The relay device 30 transfers image data and audio data to the transmission terminal 10 or the information processing terminal 40 associated with the same conference ID.

[0149] Meanwhile, the transmitter / receiver 51 of the transmission management device 50 transmits terminal status change information (operation status change information) including the communication ID "01aa" and the operation status "online (transmitting)" of the transmission terminal 10aa to each transmission terminal 10 registered in the terminal management DB 5002 (step S48). As a result, each transmission terminal that receives the terminal status change information can store and display the operation status of the transmission terminal 10aa with the communication ID "01aa" as necessary.

[0150] By establishing a session, the transmitting terminal 10aa can display image data and output audio data transmitted from the information processing terminal 40bd. The information processing terminal 40bd can display image data and output audio data transmitted from the transmitting terminal 10aa. Even if there are three or more transmitting terminals 10 participating in the video conference, the image data and audio data are transmitted in the same manner as in the case of two terminals. Alternatively, the image data and audio data may be transmitted using a communication protocol that does not involve the relay device 30 when there are two terminals, and that involves the relay device 30 when there are three or more terminals.

[0151] <Recording operations> The operation when the information processing terminal 40 performs recording will be described with reference to Fig. 10. Fig. 10 is an example of a sequence diagram showing the operation when the information processing terminal 40bd starts and ends recording.

[0152] S1: A user (conference participant) of the information processing terminal 40bd during a video conference performs an operation to record image data. The operation input receiving unit 12 of the information processing terminal 40bd receives this operation. First, the transmitting / receiving unit 11 of the information processing terminal 40bd transmits a recording start notification to the transmission management device 50 together with the communication ID (04bd) of the information processing terminal 40bd.

[0153] S2: The transmitting / receiving unit 51 of the transmission management device 50 receives the recording start notification. As a result, the status management unit 53 sets the operation status associated with the communication ID (04bd) of the information processing terminal 40bd in the terminal management table to “recording conference in progress.”

[0154] In addition, the state management unit 53 sets the operation state of all the transmitting terminals 10 and the information processing terminals 40 that are holding a video conference with the information processing terminal 40bd to "recording conference in progress." This is to inform the participants of the transmitting terminals 10 and the information processing terminals 40 who will soon participate in the video conference between the transmitting terminal 10aa and the information processing terminal 40bd that a destination terminal is participating in the conference in which image data and audio data are being recorded, since there is a risk that the participants and materials of all the transmitting terminals 10 and the information processing terminals 40 who participate in the same conference may be recorded.

[0155] The transmitting terminal 10 and the information processing terminal 40, which are holding a video conference with the information processing terminal 40bd, are registered in the session management table. The terminal extraction unit 54 extracts the conference ID associated with the communication ID (04bd) of the information processing terminal 40bd in the session management table, and further acquires all the communication IDs associated with this conference ID. In the session management table of Table 5, the transmitting terminals 10aa and 10ba, which have a communication ID of 01aa, are participating in the same conference as the information processing terminal 40bd. Therefore, the status management unit 53 sets the operating status of the information processing terminal 40bd and the transmitting terminals 10aa and 10ba to "recording conference in progress".

[0156] The state management unit 53 registers the communication ID (04bd) of the information processing terminal 40bd in the recording management table 5006. The communication IDs of the transmission terminals 10aa and 10ba do not need to be registered (because they are not recording).

[0157] S3: The transmitting / receiving unit 51 of the transmission management device 50 transmits a recording start notification to the transmitting terminal 10 and the information processing terminal 40 which are holding a video conference with the information processing terminal 40bd. The method of determining the transmitting terminal 10 and the information processing terminal 40 which are holding a video conference with the information processing terminal 40bd is the same as that in step S2. The information processing terminal 40bd and the transmitting terminals 10aa and 10ba are identified, but since the transmitting terminal 10ba is omitted in FIG. 10, the transmitting / receiving unit 51 of the transmission management device 50 transmits a recording start notification to the transmitting terminal 10aa.

[0158] S4: The transmitting / receiving unit 11 of the transmission terminal 10aa receives the recording start notification, and the display control unit 17 displays the recording notification. Details will be described with reference to FIG.

[0159] S5: When the operation input receiving unit 12 receives an operation to start recording, the image recording unit 41 of the information processing terminal 40bd starts recording image data and audio data.

[0160] S6: Next, the user of the information processing terminal 40bd (a participant of the conference) performs an operation to end the recording of the image data. The operation input acceptance unit 12 of the information processing terminal 40bd accepts this operation. Note that the image recording unit 41 of the information processing terminal 40bd ends the recording of the image data and the audio data when the operation input acceptance unit 12 accepts the operation to end the recording.

[0161] S7: The transmitter / receiver 11 of the information processing terminal 40bd transmits to the transmission management device 50 a recording end notification (an example of a recording end notification) together with the communication ID (04bd) of the information processing terminal 40bd.

[0162] S8: The transmitting / receiving unit 51 of the transmission management device 50 receives the recording end notification. The status management unit 53 deletes the communication ID (04bd) of the information processing terminal 40bd from the recording in progress management table 5006. Then, if there is no communication ID in the recording in progress management table 5006, the status "recording conference in progress" in the terminal management table is set to "online (transmitting)". In this way, if even one recording-in-progress transmitting terminal 10 or information processing terminal 40 remains, the operating status of all transmitting terminals 10 or information processing terminals 40 participating in the conference can be kept as "recording conference in progress".

[0163] S9: The transmitter / receiver 51 of the transmission management device 50 transmits a recording end notification to the transmission terminal 10 and the information processing terminal 40 that are holding a video conference with the information processing terminal 40bd. Therefore, the recording end notification is transmitted to the transmission terminal 10aa (the transmission terminal 10ba is omitted in FIG. 10).

[0164] S10: The transmitter / receiver 11 of the transmission terminal 10aa receives the recording end notification, and the display controller 17 hides the recording notification. Details will be described with reference to FIG.

[0165] In addition, the recording end notification is sent when the user performs an operation to end recording, but the recording end notification may also be sent when an indirect operation to end recording (for example, the power of the information processing terminal 40bd is turned off) is performed.

[0166] Also, the information processing terminal 40bd may transmit a recording start notification and a recording end notification directly to the transmission terminal 10aa without going through the transmission management device 50. Also, the recording start notification is sent at the start of recording, but a notification such as a recording in progress notification may be sent periodically or irregularly during the start of recording after the start of recording.

[0167] <Display example> The screen displayed on the display 120 by the transmission terminal 10 will be described.

[0168] <<Destination list screen>> Fig. 11 shows an example of a destination list screen displayed in step S32 of Fig. 8. The destination list screen displays the operating status and terminal name based on the destination status information transmitted in step S28. For convenience of explanation, the destination list screen displayed by the transmission terminal 10aa based on the terminal management table of Table 3(b) will be explained here. Note that in step S28 of Fig. 8, the transmission terminal 10aa is about to participate in a video conference, so the terminal management table of Table 3(b) does not match the processing content. Fig. 11 is a diagram for explaining that the destination list screen is created based on the terminal management table.

[0169] In the destination list management table, the communication IDs of the destination terminals of the transmission terminal 10aa are "01ab, 01ac, 01ba, 01bb, 01bc, 04bd, 04be, 01ca, 01cb, 01ca, 01da, 01db, 01dc." Of these, the information processing terminal 40bd with the communication ID 04bd and the transmission terminal 10ba with the communication ID 01ba are in the recording conference, so the destination list display unit 16 displays the recording location mark 71 in the operating status of the communication IDs 04bd and 01ba. The other transmission terminals 10 and the information processing terminals 40 are offline, so the destination list display unit 16 displays the offline mark in the operating status.

[0170] In this way, each of the destination candidates displays its operating status (recording location mark 71), so users who are about to join the video conference can understand that they and their materials will be recorded if they join the video conference. As a result, they can decide not to participate.

[0171] 12 is an example of a diagram illustrating a confirmation dialog screen 510 that is displayed when the user selects a destination with a recording location mark 71. When the user selects a destination with a recording location mark 71 displayed, the operation input receiving unit 12 receives the selection. When a destination with a recording location mark 71 displayed is selected, the destination list display unit 16 displays the confirmation dialog screen 510.

[0172] A confirmation dialog screen 510 displays a message 511 saying "You will join a video conference that is currently being recorded. Is this OK?", an OK button 512, and a Cancel button 513. When the user presses the OK button 512, the start request information of step S41 in Fig. 9 is transmitted, and when the user presses the Cancel button 513, the start request information is not transmitted. This allows the user to confirm in advance before participating in a video conference that is currently being recorded.

[0173] <<Screen during a meeting>> In this embodiment, the display control unit 17 has two display modes: a full screen display mode and an information display mode.

[0174] Fig. 13 is a diagram showing an example of a screen displayed on the display 120 in full screen display mode. This screen is called a full screen display screen 501. The full screen display mode is displayed when a session is established in step S47 in Fig. 9 and transmission or reception of image data and audio data is started. In other words, of the two display modes, this is the display mode that is primarily used during a video conference.

[0175] In the full screen display mode, the display control unit 17 of the transmitting terminal 10 displays all images transmitted from each transmitting terminal 10 participating in the video conference on the display 120. If one image area becomes small by displaying all the images, a predetermined upper limit (maximum value) of images is displayed.

[0176] 13, a relatively large first image area 502 and a plurality of relatively small second image areas 503 are allocated to a full-screen display screen 501. For example, an image represented by image data transmitted from each transmitting terminal 10 participating in the video conference is displayed in each second image area 503, and an image represented by image data of a main transmitting terminal 10 or information processing terminal 40 among the transmitting terminals 10 and information processing terminals 40 participating in the video conference is displayed in the first image area 502.

[0177] The display control unit 17, for example, identifies the transmitting terminals 10 participating in the video conference that are speaking in the received voice data as the main terminals. The transmitting terminals 10 that are speaking are sorted in descending order of volume. The display control unit 17 assigns and displays images in the first image area 502 in this order.

[0178] Note that the full-screen display screen 501 shown in FIG. 13 is merely an example, and in full-screen display mode, it is possible to display multiple images with all image areas of the same size, or to display only the image of the main terminal mentioned above across the entire display 120.

[0179] Full screen display screen 501 has auxiliary area 173. In full screen display mode, auxiliary area display unit 18a can display auxiliary area 173. In Fig. 13, auxiliary area 173 is a strip-shaped blank area at the top of full screen display screen 501, but it may be a strip-shaped blank area at the right edge, left edge, or bottom, or may be displayed so as to overlap with the image.

[0180] The auxiliary region 173 is divided into a first region ar1 to a sixth region ar6. The first area ar1 displays the communication ID of the terminal. A user of the transmitting terminal 10 who wishes to participate in the same video conference as the transmitting terminal 10 can participate in the video conference by inputting or selecting the communication ID into the transmitting terminal 10 that is currently inquiring. Second area ar2: The number of locations is displayed. Third area ar3: Displays the conference time since the session was established. Fourth area ar4: Any information is displayed. Fifth area ar5: Network bandwidth information (upstream and downstream communication speeds) is displayed. Sixth area ar6... displays the cursor button. The cursor button 1736 has an upper triangle button 1736u and a lower triangle button 1736d, with the upper triangle button 1736u being black and the lower triangle button 1736d being white. The upper triangle button 1736u indicates that it can be operated, while the lower triangle button 1736d indicates that it cannot be operated. When the upper triangle button 1736u is operated, the simplified display auxiliary area 173, which will be described later, is displayed.

[0181] It is preferable that the auxiliary area 173 is displayed semi-transparently, so that visibility can be easily maintained even if the auxiliary area 173 overlaps with image data transmitted between locations.

[0182] Here, the cursor button 1736 in the sixth area ar6 is an icon that notifies the user that it is possible to switch to the simplified display auxiliary area 173. Hereinafter, the auxiliary area in Fig. 13 is referred to as the "standard auxiliary area" to distinguish it from the "simplified display auxiliary area" described later.

[0183] The cursor button 1736 corresponds to two of the several operation buttons 108. It is preferable that the corresponding operation buttons 108 can be swung up and down, and the upward operation of the operation button 108 corresponds to the upper triangular button 1736u, and the downward operation of the operation button 108 corresponds to the lower triangular button 1736d.

[0184] Furthermore, when the auxiliary area display unit 18a receives a recording start notification, it expands the auxiliary area 173 to display the recording notification information 80. It is preferable that the recording notification information 80 is also displayed semi-transparently. In FIG. 13, the lower right portion of the auxiliary area 173 is expanded, and the other location recording mark 72 is displayed in the recording notification information 80. Also, a label 73 with the words "Recording at other location" is displayed to the right of the other location recording mark 72. It is sufficient that at least one of the other location recording mark 72 and the label 73 is displayed. The other location recording mark 72 or the label 73 allows the participants to know that image data is being recorded at another location. As is clear from the processing in FIG. 10, when one or more of the participating locations are recording, the auxiliary area display unit 18a displays the recording notification information 80.

[0185] Although the illustrated other site recording mark 72 is a red or orange circle, the appearance (color, shape, size, etc.) of the other site recording mark 72 may be, for example, a polygon such as a triangle or a square, and the color may be blue or green. The size may also be smaller or larger than that illustrated. The location of the recording notification information 80 may be the lower right part, the lower left part, the upper right part, the upper left part, or above or below the center of the auxiliary area 173. If there is space in the auxiliary area 173, at least one of the other site recording mark 72 and the label 73 may be placed in that space.

[0186] Fig. 14 is a diagram showing an example of the simplified display auxiliary area 173. In Fig. 14, the number of locations is the same as in Fig. 13, but the information on the transmission status in the first area ar1 to the sixth area ar6 is simply displayed. In the simplified display auxiliary area 173, information on what is displayed in each area is not displayed, and mainly, only the communication ID value is displayed in the first area ar1, only the number of locations in the second area ar2, only the time in the third area ar3, and only the bandwidth in the fifth area ar5.

[0187] The display form of the cursor button 1736 in the sixth area ar6 is different from that in the standard auxiliary area 173. In the simplified display auxiliary area 173, both the upper triangle button 1736u and the lower triangle button 1736d are black. The user can operate both the operation button 108 corresponding to the upper triangle button 1736u and the operation button 108 corresponding to the lower triangle button 1736d.

[0188] By operating the cursor button 1736, the display in the auxiliary area 173 is controlled as follows. Down triangle button: Switch to standard auxiliary area 173. Upper triangle button: Do not display the simplified display auxiliary area 173 (and do not display the standard auxiliary area 173 either).

[0189] The simplified display auxiliary area 173 occupies a smaller area of ​​the display 120 than the standard auxiliary area 173, so that the visibility of the image data can be improved while still displaying information about the transmission status in the auxiliary area 173.

[0190] It is preferable that the simplified display auxiliary area 173 is displayed semi-transparently, like the standard auxiliary area 173.

[0191] The display mode of the other location recording mark 72 or label 73 in the simplified display auxiliary area 173 is the same as that in the standard auxiliary area 173. However, the auxiliary area display unit 18a may change the display mode, such as by hiding either the other location recording mark 72 or the label 73. In this way, the recording notification information 80 is always displayed in the standard auxiliary area 173 and the simplified display auxiliary area 173, so that the participants can grasp the recording status in real time.

[0192] 15 is a diagram showing an example of how the other base recording mark 72 and label 73 are displayed when the auxiliary area 173 is not displayed. Even when the auxiliary area 173 is not displayed, the auxiliary area display unit 18a displays the recording notification information 80. That is, since whether or not another base is recording is important information, the auxiliary area display unit 18a always displays the recording notification information 80 even when the auxiliary area 173 is not displayed. Therefore, participants can grasp the recording status in real time.

[0193] FIG. 16 shows an example of the full-screen display screen 501 when none of the participating locations are recording. Note that FIG. 16 describes the case where the standard auxiliary area 173 is displayed. As shown in FIG. 16, when there are no more locations recording, the auxiliary area display unit 18a hides (deletes) the recording notification information 80. This allows the participants to know in real time that there are no more locations recording. Note that the same is true for the simplified display auxiliary area 173, where the recording notification information 80 is not displayed. If the auxiliary area 173 is not displayed, the recording notification information 80 is not displayed either.

[0194] <<Display control of recording notification information during a meeting>> Fig. 17 is an example of a flowchart showing a procedure in which the transmission terminal 10aa during a conference controls the display of the recording notification information 80. Since the recording notification information 80 is always displayed on the full-screen display screen 501, the procedure in Fig. 17 does not require the auxiliary area to be displayed.

[0195] First, the auxiliary area display unit 18a determines whether or not a recording start notification or a recording end notification has been received (S10).

[0196] When the recording start notification is received, the auxiliary area display unit 18a adds the communication ID transmitted together with the recording start notification to the recording management table 1000b (S20). Therefore, the recording management table 1000b records all the communication IDs of the transmitting terminal 10 and the information processing terminal 40 that are recording.

[0197] The auxiliary area display unit 18a determines whether or not the recording notification information 80 is already displayed (S30). The case where the recording notification information 80 is already displayed means that after the transmitting terminal 10 or the information processing terminal 40 participating in the video conference has transmitted a recording start notification, another transmitting terminal 10 or information processing terminal 40 has transmitted a recording start notification.

[0198] If the determination in step S30 is Yes, the recording notification information 80 is already being displayed, so the process in FIG. 17 ends.

[0199] If the determination in step S30 is No, the auxiliary area display unit 18a displays the recording notification information 80 (S40).

[0200] When the recording end notification is received in step S10, the auxiliary area display unit 18a deletes the communication ID transmitted together with the recording start notification from the recording management table 1000b (S50). Therefore, only the communication IDs of the transmission terminal 10 and the information processing terminal 40 that are continuing recording are recorded in the recording management table 1000b.

[0201] The auxiliary area display unit 18a determines whether or not any communication ID is registered in the recording management table 1000b (S60).

[0202] If the determination in step S60 is No, there is no need to hide the recording notification information 80 since there is the transmission terminal 10 and the information processing terminal 40 that are continuing recording, and the process in FIG. 17 ends.

[0203] If the determination in step S60 is Yes, there is no transmission terminal 10 or information processing terminal 40 continuing recording, and therefore the auxiliary area display unit 18a does not display the recording notification information 80 (S70).

[0204] With this procedure, if the transmitting terminal 10 and the information processing terminal 40 that have started recording are present, the recording notification information 80 can be displayed in real time, and if the transmitting terminal 10 and the information processing terminal 40 that are continuing recording are no longer present, the recording notification information 80 can be hidden. Also, if at least one communication ID is stored in the recording in progress management table 1000b, the recording notification information 80 can always be displayed.

[0205] <<If a network disconnection occurs>> During a conference, a session may be interrupted due to a decrease in network bandwidth or the like. Or, a participant may accidentally pull out a cable required for communication. If the transmission terminal 10 or information processing terminal 40 whose session has been interrupted is recording, the transmission terminal 10 or information processing terminal 40 at the other base will continue to display the recording notification information 80. For this reason, it is preferable that the transmission management device 50 detects the transmission terminal 10 or information processing terminal 40 whose session has been interrupted and notifies the transmission terminal 10 or information processing terminal 40 at the other base even if the recording end information is not transmitted.

[0206] 18 is an example of a sequence diagram showing an operation procedure of the transmission system 1 when the transmission management device 50 detects that a session has been disconnected. Note that the processing from steps S1 to S5 in FIG. 18 is the same as that in FIG.

[0207] S6: The session management unit 56 of the transmission management device 50 periodically monitors the alive status of the transmission terminal 10 and the information processing terminal 40 with which the session has been established. That is, it requests a Ping response or the like from the transmission terminal 10 and the information processing terminal 40 of the communication ID registered in the session management table.

[0208] S7: If the communication between the information processing terminal 40 and the transmission management device 50 is cut off for some reason, the session management unit 56 can detect that the session has been cut off.

[0209] S8: When the session management unit 56 detects that the session has been interrupted, the status management unit 53 changes the operation status of the communication ID of the information processing terminal 40bd whose session has been interrupted to offline. The status management unit 53 deletes the communication ID of the information processing terminal 40bd from the recording in progress management table 5006, and updates the operation status in the terminal management table in the same manner as when a recording end notification is received.

[0210] S9: The transmitting / receiving unit 51 of the transmission management device 50 transmits operating status change information of the participating locations to the transmission terminal 10 and the information processing terminal 40 in the conference. The terminal status change information includes the communication ID and operating status of the information processing terminal 40bd.

[0211] S10: The transmitter / receiver 11 of the transmission terminal 10aa receives the terminal state change information and controls the display of the recording notification information 80. That is, if the communication IDs of the transmission terminal 10 and the information processing terminal 40 other than the information processing terminal 40bd whose session has been interrupted are not registered in the recording management table 1000b, the recording notification information 80 is hidden.

[0212] The process of step S10 in Fig. 18 will be described with reference to Fig. 19. Fig. 19 is an example of a flowchart illustrating the process when the transmission terminal 10aa receives terminal state change information. The process in Fig. 19 is repeatedly executed during the video conference.

[0213] The transmitting / receiving unit 11 of the transmission terminal 10aa determines whether or not the terminal state change information has been received from the transmission management device 50 (S10).

[0214] If the determination in step S10 is Yes, the auxiliary area display unit 18a of the transmitting terminal 10aa checks the transmitting terminals 10 and information processing terminals 40 that have left the room (become offline) (S20). That is, since the transmitting terminal 10aa has received the destination status information and the terminal status change information, it refers to the operation status of each transmitting terminal 10 or information processing terminal 40 in the destination status information and terminal status change information that it has received last, and compares it with the terminal status change information received in step S10. Through the comparison, it identifies the communication ID of each transmitting terminal 10 or information processing terminal 40 whose operation status has changed from "recording conference" to "offline."

[0215] Then, the auxiliary area display unit 18a determines whether the identified communication ID exists in the recording location ID table (S30).

[0216] If the communication ID does not exist in the recording location ID table (No in S30), the information processing terminal 40bd whose session has been interrupted has not been recording, and the process in FIG. 19 ends.

[0217] If the communication ID exists in the recording location ID table (Yes in S30), since the information processing terminal 40bd whose session was interrupted was recording, the auxiliary area display unit 18a deletes the communication ID included in the terminal status change information from the recording management table 1000b (S40).

[0218] The auxiliary area display unit 18a determines whether or not any communication ID is registered in the recording in progress management table 1000b (S50).

[0219] If the determination in step S50 is No, there is no need to hide the recording notification information 80 since there is the transmission terminal 10 and the information processing terminal 40 that are continuing recording, and the process in FIG. 19 ends.

[0220] If the determination in step S50 is Yes, there is no transmission terminal 10 or information processing terminal 40 continuing recording, and therefore the auxiliary area display unit 18a does not display the recording notification information 80 (S70).

[0221] By this procedure, even if the participating locations are changed, the transmission terminal 10aa can update the recording notification information 80 in real time.

[0222] <Recording notification information on the information display screen> The recording location mark 71 displayed on the information display screen 601 will be described. The user can alternate between two display modes by operating, for example, the operation button 108. When the transmission terminal 10 is the information processing terminal 40, the user can alternate between two display modes by pressing a hard key of the information processing terminal 40 or a soft key displayed on a touch panel instead of the operation button 108.

[0223] Fig. 20 is a diagram showing an example of an information display screen 601 displayed on the display 120 in the information display mode. Fig. 20 shows the information display screen 601 displayed by the transmitting terminal 10aa. As shown in Fig. 20, in the information display mode, the location list display unit 18b of the transmitting terminal 10 displays terminal information including a serial number 614, a terminal name 615, and a communication ID 616 of each transmitting terminal 10 participating in the video conference on the display 120. The serial number 614 is a number assigned to each terminal in the order of participation, and the largest number in the serial number 614 indicates the number of locations (number of terminals) participating.

[0224] Also, the information processing terminal 40bd displays a recording location mark 71. As will be described later, the location list display unit 18b displays the recording location mark 71 in front of the serial numbers of the transmission terminal 10 and the information processing terminal 40 whose operating status is recording a conference, as shown in the figure. The display mode of the recording location mark 71 is not limited to the illustrated one, like the other location recording mark 72.

[0225] 20, an image display area 520 for displaying an image and a terminal information display area 530 for displaying terminal information are allocated. For example, the terminal information of the transmission terminals 10 participating in the video conference is displayed in a list in the terminal information display area 530.

[0226] Among the terminal information displayed in the list, those without a terminal name indicate transmission terminals 10 that are not registered as candidates for the destination terminal of the own terminal in the destination list management DB 5003. For example, the transmission terminal 10db with the communication ID 01db is registered in the destination list management DB 5003 as a candidate for the destination terminal of the transmission terminal 10aa as shown in Table 3, and therefore the terminal name is displayed. The transmission terminal 10cd with the communication ID 01cd is not registered in the destination list management DB 5003 as a candidate for the destination terminal of the transmission terminal 10aa, and therefore the terminal name is not displayed.

[0227] An image represented by image data of a major transmitting terminal among the transmitting terminals 10 participating in the video conference is displayed in the image display area 520. For example, image data of the transmitting terminal 10 or the information processing terminal 40 having the loudest audio data volume is displayed.

[0228] 20, if the locations participating in the conference do not fit on one screen, a scroll bar 613 is displayed. The user can display information about a location hidden outside the screen of the information display screen 601 by operating the operation button 108. As a result, if a location hidden outside the screen is recording, a recording location mark 71 is displayed, and the participant can identify the transmission terminal 10 or the information processing terminal 40 that is recording.

[0229] (Display procedure of information display screen 601) Fig. 21 is a sequence diagram showing a terminal information display operation for displaying terminal information in the terminal information display area 530 when the transmission terminal 10aa is switched to the information display mode. In the terminal information display operation shown in Fig. 21, various information is transmitted and received through the management information session sei.

[0230] First, the transmission terminal 10aa transmits a terminal information request signal representing a request for terminal information of the transmission terminals and the information processing terminal 40 participating in the video conference to the transmission management device 50 (step S70). That is, when the user operates the operation button 108 to switch the full screen display mode to the information display mode, the operation input receiving unit 12 receives this operation and transmits the terminal information request signal to the transmission management device 50.

[0231] The terminal status acquisition unit 55 of the transmission management device 50 that has received the terminal information request signal searches the terminal management DB 5002 of the storage unit 5000 using the IP address of the sender of the terminal information request signal as a search key, and acquires the communication ID of the transmission terminal 10aa that sent the terminal information request signal (step S71). Note that the transmission terminal 10aa may send the communication ID.

[0232] Next, the terminal status acquisition unit 55 refers to the session management table to identify the communication IDs of the transmission terminal 10 and the information processing terminal 40 participating in the same conference as the transmission terminal 10aa. Next, the terminal status acquisition unit 55 searches the terminal management DB 5002 to search for communication IDs and terminal names whose operating status is "online (transmitting)" or "recording conference" (step S72).

[0233] Next, the terminal status acquisition unit 55 searches the destination list management DB 5003 in the memory unit 5000 using the communication ID detected in step S71 (in this embodiment, "01aa") as a search key, and acquires the communication IDs of the transmission terminal 10 and the information processing terminal 40 that have the source of the terminal information request signal as the destination terminal (step S73).

[0234] Next, the terminal status acquisition unit 55 judges whether each communication ID acquired in step S72 is included in the communication ID (01aa) of the sender of the terminal information request signal or the communication ID acquired in step S73. If it is judged that the communication ID and the terminal name are included, the transmission / reception unit 51 responds with the communication ID and the terminal name to the sender of the terminal information request signal, and if it is judged that the communication ID and the terminal name are not included, the transmission / reception unit 51 responds with only the communication ID to the sender of the terminal information request signal (step S74).

[0235] The location list display unit 18b of the display control unit 17 of the transmission terminal 10 that receives this response displays the response content in the terminal information display area 530 as in the information display screen 601 shown in Fig. 20 (step S75). First, a consecutive number is assigned to the set of the terminal name (there may be no terminal name) and the communication ID. Next, all communication IDs in the recording management table 1000b are acquired. Then, the communication ID in the recording management table 1000b is read, and a recording location mark 71 is arranged in association with the transmission terminal 10 or the information processing terminal 40 of this communication ID.

[0236] By such a process, the recording location mark 71 can be displayed on the information display screen 601. In addition to displaying the recording location mark 71, or without displaying the recording location mark 71, the transmitting terminal 10aa may read out the name of the terminal that is recording by voice. In this case, the transmitting terminal 10aa reads out the name of the terminal that is recording by voice at the timing when the recording start notification is transmitted. Alternatively, the transmitting terminal 10aa reads out the name of the terminal that is recording by voice at any timing when the user operates the transmitting terminal 10aa.

[0237] In addition, when the transmission terminal 10aa receives a recording start notification, a recording end notification, or terminal status change information while the information display screen 601 is being displayed, the location list display unit 18b of the display control unit 17 displays or hides the recording location mark 71 in real time.

[0238] Furthermore, not only the recording location mark 71 but also the label 73 may be displayed on the information display screen 601. Furthermore, only the label 73 may be displayed on the information display screen 601.

[0239] Therefore, while the transmission terminal 10aa is holding a video conference with multiple locations, the user of the transmission terminal 10aa can determine whether or not recording is in progress for each location (the transmission terminal 10 and the information processing terminal 40) that is recording. Since the recording location mark 71 is updated in real time, the user can determine the location that has started recording soon after starting recording. The user can know that there is a location that is recording on both the full screen display screen 501 and the information display screen 601. EXAMPLES

[0240] In this embodiment, an electronic whiteboard 8 equipped with the functions of a transmission terminal 10 will be described. Conventionally, electronic whiteboards 8 capable of drawing stroke images such as letters, numbers, and figures on a background image displayed on a large display are known. Electronic whiteboards 8 are widely used in companies, educational institutions, government agencies, and other places.

[0241] The electronic whiteboard 8 may have a function of superimposing an image input from the outside on the display. Also, it is being considered to equip the electronic whiteboard 8 with the function of the transmission terminal 10. When the electronic whiteboard 8 is equipped with the function of the transmission terminal 10, a stroke image handwritten on the electronic whiteboard 8 can be transmitted to the transmission terminal 10 at another site.

[0242] 22 is an example of a diagram for explaining an electronic whiteboard 8 having the functions of the transmission terminal 10. The electronic whiteboard 8 has the functions of a general electronic whiteboard and the functions of the transmission terminal 10, and can hold a video conference with the transmission terminal 10dd. The electronic whiteboard 8 switches between displaying a stroke image of an electronic whiteboard application and an image of a video conference application.

[0243] However, in the electronic whiteboard 8, most of the display 3 is occupied by stroke images, making it difficult to see the status change of a video conference app, for example, which is running but not displayed on the screen. This causes a problem that the user U2 has difficulty noticing the status change of the video conference app.

[0244] For example, a stroke image generated by an electronic whiteboard application of the electronic whiteboard 8 is transmitted to the transmission terminal 10dd by a video conference application. A user U1 of the transmission terminal 10dd can start recording as in the first embodiment. However, even if the video conference application of the electronic whiteboard 8 receives a recording start notification from another location, a user U2 of the electronic whiteboard 8 may not notice it. For this reason, there is a risk that private information may be recorded without being noticed.

[0245] 22, "confidential information" handwritten on the electronic whiteboard 8 is shared with the transmission terminal 10dd. Since the user U1 of the transmission terminal 10dd is recording image data, the video conference application of the electronic whiteboard 8 is displaying the recording location mark 71 or the other location recording mark 72, but the user U2 cannot see it. Recording such non-public information may, for example, cause the risk of insider fraud or incidents, or cause some inconvenience in communication.

[0246] Therefore, in this embodiment, a transmission system 1 is described in which, when a recording start notification is sent to an electronic whiteboard 8 having the functions of a transmission terminal 10, the electronic whiteboard application displays information of a video conferencing application 710 on a display 3, thereby notifying a user U2.

[0247] <Outline of the operation of the electronic whiteboard 8> Fig. 23 is an example of a diagram for explaining an outline of the operation of the electronic whiteboard 8 of this embodiment. As shown in Fig. 23(a), the electronic whiteboard 8 has a video conference application 710, an electronic whiteboard application 720, and an interface application 730. These are application software that run on an OS (Operating System) 700. Any appropriate OS 700 may be used, and there is no requirement that it be a specific OS 700, but examples of such OS include Windows (registered trademark), Linux (registered trademark), and Android (registered trademark).

[0248] The video conference application 710 provides the functions of the transmission terminal 10 or the information processing terminal 40 described in the first embodiment. The functions of the electronic whiteboard application 720 will be described later.

[0249] 23(b) is a diagram explaining the operation of the interface application 730. The interface application 730 is an application that provides an interface for two independent applications: a video conference application 710 and an electronic whiteboard application 720. To communicate with the interface application 730, the video conference application 710 has an interface processing unit 710a, and the electronic whiteboard application 720 has an interface processing unit 720a.

[0250] The interface application 730 communicates with the video conference application 710 via the interface processing unit 710a, and communicates with the electronic whiteboard application 720 via the interface processing unit 720a. For example, the interface application 730 detects that the video conference application 710 has received a recording start notification, or that the video conference is being recorded, or detects the recording management table, and notifies the electronic whiteboard application 720. This allows the electronic whiteboard application 720 to display that another location is recording.

[0251] <Example of the display screen of the electronic whiteboard 8> 24 is an example of a diagram showing the display 3 of the electronic whiteboard 8 on which an electronic whiteboard UI screen 740 generated by the electronic whiteboard application 720 is displayed. The electronic whiteboard UI screen 740 is a screen (second screen) on which mainly a stroke image handwritten by a user is displayed. In addition, the electronic whiteboard UI screen 740 has an icon section 701, and the user can select a line type and color of a stroke, and draw a figure or a stamp.

[0252] A video conference icon 702a is displayed in this icon section 701. When a user selects the video conference icon 702a, the electronic whiteboard 8 displays a video conference UI screen 750 (described below) generated by a video conference application 710 on the display 3. Note that such processing corresponds to switching between applications, and is controlled by the OS 700.

[0253] FIG. 25 is an example of a diagram showing the display 3 of the electronic whiteboard 8 on which a video conference UI screen 750 generated by the video conference application 710 is displayed. The video conference UI screen 750 has an electronic whiteboard icon 703 (first screen) in addition to information equivalent to that of the destination list screen in FIG. 11. When a user selects the electronic whiteboard icon 703, the electronic whiteboard 8 displays an electronic whiteboard UI screen 740 generated by the electronic whiteboard application 720 on the display 3. Note that such processing corresponds to switching between applications, and is controlled by the OS. Therefore, the user can arbitrarily switch between the video conference UI screen 750 and the electronic whiteboard UI screen 740.

[0254] Note that, instead of displaying either the video conference UI screen 750 or the electronic whiteboard UI screen 740, both the video conference UI screen 750 and the electronic whiteboard UI screen 740 may be displayed with a partial overlap. In this case, when the other site recording mark on the video conference UI screen 750 is hidden by the electronic whiteboard UI screen 740, this embodiment is effective.

[0255] <Hardware configuration of Electronic Whiteboard 8> Fig. 26 is a hardware configuration diagram of the electronic whiteboard 8. As shown in Fig. 26, the electronic whiteboard 8 includes a CPU 301 that controls the overall operation of the electronic whiteboard 8, a ROM 302 that stores programs used to drive the CPU 301, such as an IPL, a RAM 303 that is used as a work area for the CPU 301, an SSD 304 that stores various data, such as programs for the electronic whiteboard 8, a network controller 305 that controls communication with the transmission network 2, and an external storage controller 306 that controls communication with the USB memory 5.

[0256] The electronic whiteboard 8 also includes a capture device 309 that displays video information as still or moving images on the display 3 of the notebook PC 6, a GPU (Graphics Processing Unit) 308 that specializes in graphics, and a display controller 313 that controls and manages the screen display to output images from the GPU 308 to the display 3, etc.

[0257] Furthermore, the electronic whiteboard 8 includes a sensor controller 307 that controls the processing of the contact sensor 312, and the contact sensor 312 that detects the contact of the electronic pen 4, the user's hand H, or the like on the display 3. The contact sensor 312 inputs and detects coordinates by an infrared blocking method. The method of inputting and detecting coordinates is a method in which two light receiving and emitting devices installed at both ends of the upper side of the display 3 emit a plurality of infrared rays in parallel to the display 3, and receives light that is reflected by a reflecting member installed around the display 3 and returns along the same optical path as the optical path of the light emitted by the light receiving element. The contact sensor 312 outputs ID (Identification) of the infrared rays emitted by the two light receiving and emitting devices that are blocked by an object to the sensor controller 307, and the sensor controller 307 identifies the coordinate position that is the contact position of the object.

[0258] In addition, the contact sensor 312 is not limited to an infrared blocking type, and various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by the voltage change of two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contacting object touching the display unit.

[0259] The electronic whiteboard 8 also includes an electronic pen controller 310. The electronic pen controller 310 communicates with the electronic pen 4 to determine whether or not the tip or tail of the pen has touched the display 3. The electronic pen controller 310 may be configured to determine whether or not the tip or tail of the electronic pen 4 has touched the part of the electronic pen 4 that is held by the user or other parts of the electronic pen has touched the display 3.

[0260] Furthermore, the electronic whiteboard 8 has bus lines 320 such as an address bus and a data bus for electrically connecting the CPU 301, ROM 302, RAM 303, SSD 304, network controller 305, external memory controller 306, capture device 309, GPU 308, sensor controller 307, and electronic pen controller 310 as shown in FIG. 26.

[0261] <Functional configuration of the electronic whiteboard app 720> Next, the functional configuration of the electronic whiteboard 8 will be described with reference to Fig. 27 to Fig. 32. Fig. 27 is a functional block diagram of the electronic whiteboard 8.

[0262] 26, the electronic whiteboard 8 has an image acquisition unit 21, a coordinate detection unit 22, an automatic adjustment unit 23, a contact detection unit 24, an event allocation unit 25, an operation processing unit 26, a gesture processing unit 27, an image superimposition unit 28, and an image processing unit 9. These functions surrounded by dotted lines are functions of the video conference application 710.

[0263] Among these, the video acquisition unit 21 acquires an output video of a video output device such as a notebook PC 6 connected to a cable. When the video acquisition unit 21 receives an image signal from the video output device, it analyzes the image signal to derive image information such as the resolution of an image frame, which is a display image of the video output device formed by the image signal, and the update frequency of this image frame, and outputs the information to the image acquisition unit 31 described later.

[0264] The coordinate detection unit 22 detects the coordinate position of an event (such as the action of the user's hand H touching the display 3) that occurs on the display 3 by the user. The coordinate detection unit 22 also detects the area of ​​the touch.

[0265] The automatic adjustment unit 23 is started when the interactive whiteboard 8 is started, and adjusts image processing parameters of the sensor camera in the optical sensor system of the contact sensor 312 so that the contact sensor 312 can output an appropriate value to the coordinate detection unit 22.

[0266] The contact detection unit 24 detects an event caused by the user (such as an action of pressing (touching) the display 3 with the tip or the bottom of the electronic pen 4).

[0267] The event allocation unit 25 allocates the coordinate positions of the events detected by the coordinate detection unit 22 and the detection results detected by the contact detection unit 24 to the events of stroke drawing, UI operation, and gesture operation. Here, the stroke drawing, UI operation, and gesture operation will be described with reference to Fig. 28. Fig. 28 is a configuration diagram of each image layer.

[0268] First, a "stroke drawing" is drawn by an event in which a user presses the electronic pen 4 on the display 3, moves the electronic pen 4 while keeping it pressed, and finally releases the electronic pen 4 from the display 3. By this stroke drawing, for example, alphabet characters such as "S" and "T" are drawn on the display 3. Note that this "stroke drawing" includes not only an event of drawing an image, but also an event of deleting an image that has already been drawn and an event of editing an image that has already been drawn.

[0269] A "UI operation" is an event in which the user presses a predetermined position with the electronic pen 4 or hand H when a UI image (A) shown in Fig. 28 (described later) is displayed on the display 3. This UI operation sets, for example, the color, width, etc. of a line drawn by the electronic pen 4.

[0270] A "gesture operation" is an event in which the user touches or moves the display 3 with the hand H when a stroke image (B) shown in Fig. 28 (described later) is displayed on the display 3. With this gesture operation, for example, the user can enlarge (or reduce) the image, change the display area, or switch pages by moving the hand H while touching the display 3 with the hand H.

[0271] 27, the operation processing unit 26 executes various operations according to the UI element in which the event occurred, based on the UI operation determined to be a UI operation by the event allocation unit 25. Examples of the UI element include a button, a list, a check box, and a text box.

[0272] The gesture processing unit 27 executes an operation corresponding to the operation determined by the event allocation unit 25 to be a gesture operation.

[0273] The video superimposition unit 28 displays the image superimposed by the display superimposition unit 36 ​​described below as a video on a video output device (such as the display 3). The video superimposition unit 28 also performs picture-in-picture of a video sent from another video output device (such as the camera 7) on the video from the video output device (such as the notebook PC 6). Furthermore, the video superimposition unit 28 performs switching so that a video that has been subjected to picture-in-picture and displayed on a part of the display 3 is displayed on the entire display 3.

[0274] (Functional configuration of the image processing unit) Next, a detailed description will be given of the functional configuration of the image processing unit 9. The image processing unit 9 performs processes such as superimposing each image layer as shown in FIG.

[0275] Therefore, the image processing unit 9 has an image acquisition unit 31, a stroke processing unit 32, a UI image generation unit 33, a background generation unit 34, a layout management unit 35, a display overlay unit 36, a page processing unit 37, a file processing unit 42, and a page data storage unit 300.

[0276] Of these, image acquisition section 31 acquires, as an image, each frame from the video acquired by video acquisition section 21. This image corresponds to output image (C) shown in FIG.

[0277] The stroke processing unit 32 draws an image, deletes a drawn image, or edits a drawn image based on the stroke drawing-related event allocated by the event allocation unit 25. The image by the stroke drawing corresponds to the stroke image (B) shown in FIG.

[0278] The UI image generating unit 33 generates a UI (user interface) image that is set in advance on the electronic whiteboard 8. This UI image corresponds to the UI image (A) shown in FIG.

[0279] The background generation unit 34 generates a background image to be displayed on the display 3. This background image corresponds to the background image (D) shown in Fig. 28. The pattern of the background image is a solid color, a grid display, or the like.

[0280] The layout management unit 35 manages layout information indicating the layout of each image output from the image acquisition unit 31, stroke processing unit 32, UI image generation unit 33, and background generation unit 34 for the display superimposition unit 36. This allows the layout management unit 35 to instruct the display superimposition unit 36 ​​on the position in the UI image (A) and background image (D) at which the output image (C) and stroke image (B) should be displayed or not displayed.

[0281] The display superimposition unit 36 ​​superimposes the output image (C) from the image acquisition unit 31, the stroke image (B) from the stroke processing unit 32, the UI image (A) from the UI image generation unit 33, and the background image (D) from the background generation unit 34 according to a layout specified by the layout management unit 35. As a result, as shown in Fig. 28, the layers are configured as the UI image (A), stroke image (B), output image (C), and background image (D) in an order that the user can see even when the images are superimposed.

[0282] The display superimposition unit 36 ​​can also switch between images (C) and (D) shown in Fig. 28 and superimpose them exclusively on images (A) and (B). For example, if a cable is disconnected between the electronic whiteboard 8 and a video output device (such as the notebook PC 6) while images (A), (B), and (C) are initially displayed, image (C) can be removed from the list of images to be superimposed and image (D) can be displayed, as specified by the layout management unit 35. In this case, the display superimposition unit 36 ​​also performs processes for enlarging and reducing the display and moving the display area.

[0283] The page processing unit 37 compiles the data of the stroke image (B) and the data of the output image (C) into one page data and stores it in the page data storage unit 300. In addition, the page processing unit 37 transmits the stored page data via the image acquisition unit 31 to the display superimposition unit 36, so that the video superimposition unit 28 can redisplay the page data on the display 3.

[0284] Furthermore, the page processing unit 37 can transmit the page data (stroke array data, media data) from the video conference application 710 to another electronic whiteboard 8 or a transmission terminal 10 by transmitting the stored page data to the video conference application 710 via the image acquisition unit 31. Also, the page processing unit 37 can make the page data available for re-editing strokes by transmitting the page data to the stroke processing unit 32. Furthermore, the page processing unit 37 can delete or copy the page data.

[0285] When the page processing unit 37 reads out page data from the page data storage unit 300, the page processing unit 37 separates the combined page data into stroke array data indicating a stroke image (B) and media data indicating a background image (D). That is, the data of the output image (C) displayed on the display 3 at the time when the page processing unit 37 stores the data in the page data storage unit 300 is temporarily stored in the page data storage unit 300, and when the data is subsequently read out from the page data storage unit 300, it is read out as media data indicating a background image (D). Then, the page processing unit 37 outputs the stroke array data indicating the stroke image (B) from the page data read out from the page data storage unit 300 to the stroke processing unit 32. Furthermore, the page processing unit 37 outputs the media data indicating the background image (D) from the page data read out from the page data storage unit 300 to the image acquisition unit 31.

[0286] The page data storage unit 300 stores page data as shown in Fig. 29. Fig. 12 is a conceptual diagram showing page data. The page data is data (stroke data and image data) for one page displayed on the display 3. Note that since the page data includes many types of parameters, the contents of the page data will be explained here by dividing them into Figs. 29 to 31.

[0287] After the electronic whiteboard 8 has abnormally terminated, the file processing unit 42 detects the abnormal termination and recovers the unsaved page data. For example, in the case of normal termination, the file processing unit 42 acquires the page data from the page data storage unit 300 and records it as a PDF file in the USB 5. However, in the case of abnormal termination due to a power outage or the like, the page data remains recorded in the page data storage unit 300. Therefore, when the power is turned on again, the file processing unit 42 recovers the page data by reading it from the page data storage unit 300. The file processing unit 42 also reads the PDF file from the USB memory 5 and stores each page in the page data storage unit 300 as page data.

[0288] Here, the page data will be described in detail. As shown in Fig. 29, the page data is stored in association with a page data ID for identifying any one page, a start time indicating the time when the display of this page started, an end time indicating the time when the content of the page was no longer rewritten by strokes, gestures, etc., a stroke array data ID for identifying stroke array data generated by strokes by the electronic pen 4 or the user's hand H, and a media data ID for identifying media data. The stroke array data is data for displaying a stroke image (B) shown in Fig. 11 described later on the display 3. The media data is data for displaying a background image (D) shown in Fig. 28 described later on the display 3.

[0289] With such page data, for example, when a user draws the alphabet "S" with the electronic pen 4, the alphabet [S] is represented by one stroke data ID since the character is drawn in one stroke. However, when a user draws the alphabet "T" with the electronic pen 4, the alphabet [T] is represented by two stroke data IDs since the character is drawn in two strokes.

[0290] Moreover, the stroke array data indicates detailed information as shown in Fig. 30. Fig. 30 is a conceptual diagram showing stroke array data. As shown in Fig. 30, one piece of stroke array data is represented by a plurality of stroke data. One piece of stroke data indicates a stroke data ID for identifying this stroke data, a start time indicating the time when writing of one stroke begins, an end time indicating the time when writing of one stroke ends, the color of the stroke, the width of the stroke, and a coordinate array data ID for identifying the array of passing points of the stroke.

[0291] Furthermore, this coordinate array data indicates detailed information as shown in Fig. 31. Fig. 31 is a conceptual diagram showing coordinate array data. As shown in Fig. 31, the coordinate array data indicates each piece of information on one point on the display 3 (X coordinate value, Y coordinate value), the time difference (ms) from the start time of the stroke when passing this one point, and the writing pressure of the electronic pen 4 at this one point. That is, a collection of one point shown in Fig. 31 is indicated by one piece of coordinate array data shown in Fig. 30. For example, when a user draws the alphabet "S" with the electronic pen 4, it is drawn in one stroke, but since multiple passing points are passed before the "S" is completed, the coordinate array data indicates information on these multiple passing points.

[0292] Moreover, the media data of the page data shown in Fig. 29 shows detailed information as shown in Fig. 30. Fig. 32 is a conceptual diagram showing media data. As shown in Fig. 32, the media data shows the media data ID in the page data shown in Fig. 29, the data type of the media data, the recording time when the page data was stored in the page data storage unit 300 from the page processing unit 37, the position (X coordinate value, Y coordinate value) and size (width, height) of the image displayed on the display 3 by the page data, and data showing the contents of the media data, all of which are associated with each other. Among these, the position of the image displayed on the display 3 by the page data shows the position of the upper left corner of the image displayed by the page data when the coordinates of the upper left corner of the display 3 are (X coordinate value, Y coordinate value) = (0, 0).

[0293] (Video conferencing app functional configuration) Next, a detailed description will be given of the functional configuration of the video conference application 710. The video conference application 710 controls the process for the electronic whiteboard 8 to communicate with external devices (the transmission terminal 10, the UI providing device 20, the other electronic whiteboards 8, the transmission management device 50, the relay device 30, and the program providing device 90).

[0294] For this reason, the video conference application 710 has a transmission / reception unit 61, an operation input reception unit 62, a login request unit 63, a processing unit 64, an output superimposition unit 65, a display control unit 66, a destination list creation unit 67, a determination unit 68, and a storage / read processing unit 69. Furthermore, the video conference application 710 has a storage unit 6000 constructed by the SSD 304 shown in FIG.

[0295] 6. That is, the transmitting / receiving unit 61, the operation input receiving unit 62, the login request unit 63, the display control unit 66, the destination list creating unit 67, and the storage / read processing unit 69 of the video conference application 710 perform the same processes as the transmitting / receiving unit 11, the operation input receiving unit 12, the login request unit 13, the destination list display unit 16, the display control unit 17, and the storage / read processing unit 19 of the transmitting terminal 10, respectively. Also, the storage unit 6000 of the video conference application 710 plays the same role as the storage unit 1000 of the transmitting terminal 10. Therefore, the description thereof will be omitted.

[0296] On the other hand, when the display capability information (screen size, resolution, etc. of the other transmission terminal 10) received by the transmitting / receiving unit 61 indicates that the other transmission terminal does not have the capability to display a predetermined image, the processing unit 64 processes the image data. Specifically, the processing unit 64 cuts out operation image data related to the UI image by cropping from the image data in which images of multiple layers including a UI image (A) for performing an operation on the electronic whiteboard 8 are superimposed, and then resizes the image data after cropping to enlarge it to the image data before cropping. Note that the processing unit 64 may perform cropping after resizing to enlarge the image data.

[0297] The output superimposition unit 65 superimposes image data on image data, like the display superimposition unit .

[0298] If the display capability information received by the transmitting / receiving unit 61 indicates that the other transmitting terminal does not have the capability to display a specific image, the determining unit 68 determines that the image data is to be processed. As a result, the processing unit 64 processes the image data.

[0299] (Interface App 730) As shown in Fig. 27, the electronic whiteboard 8 has an interface unit 730a. The interface unit 730a is a function or means realized by the CPU 301 of the electronic whiteboard 8 executing the interface application 730 and cooperating with various hardware shown in Fig. 26. The interface unit 730a monitors for a certain period of time (for example, between one second and one minute) whether or not "a recording start notification has been received" is stored in the storage / read processing unit 69 of the video conference application 710 via the interface processing unit 710a. In addition, the interface unit 730a transmits a signal to the OS 700 to instruct the application to be displayed in the foreground of the display 3.

[0300] The video conference application 710 also has an interface processor 710a. The interface processor 710a is a function or means realized by the CPU 301 of the electronic whiteboard 8 executing the video conference application 710 and cooperating with various hardware components shown in Fig. 26. When the electronic whiteboard icon 703 is pressed, the interface processor 710a transmits an electronic whiteboard UI screen switching signal indicating that the electronic whiteboard application 720 has been selected to the interface unit 730a. As a result, the interface unit 730a causes the OS 700 to display the electronic whiteboard UI screen 740.

[0301] The electronic whiteboard application 720 also has an interface processor 720a. The interface processor 720a is a function or means realized by the CPU 201 of the electronic whiteboard 8 executing the electronic whiteboard application 720 and cooperating with various hardware components shown in Fig. 26. When the video conference icon 702a is pressed, the interface processor 720a transmits a video conference UI screen switching signal indicating that the video conference application 710 has been selected to the interface unit 730a. As a result, the interface unit 730a causes the OS 700 to display the video conference UI screen 750.

[0302] <Operation procedure> Fig. 33 is an example of a sequence diagram showing an operation procedure when the video conference application 710 running on the interactive whiteboard 8 receives a recording start notification. In Fig. 33, it is assumed that a video conference has already been held by the processes in Figs. 8 and 9.

[0303] S1: A recording start notification is transmitted from the transmission terminal 10dd to the transmission management device 50. The recording start notification has been described with reference to FIG.

[0304] S2: The transmitter / receiver 51 of the transmission management device 50 transmits a recording start notification to the electronic whiteboard 8 that is holding a video conference with the transmission terminal 10dd.

[0305] S3, S4: The interface unit 730a monitors at regular time intervals whether or not a recording start notification has been received from the storage / read processing unit 69 of the video conference application 710 via the interface processing unit 710a.

[0306] S5: When a recording start notification is received, the interface unit 730a judges whether or not this is the first time that the video conference application 710 has received a recording start notification. Since the storage unit 6000 of the video conference application 710 also has a recording in progress management table, when a communication ID is registered in a state where the communication ID in the recording in progress management table is zero, the interface unit 730a judges that this is the first time that the video conference application 710 has received a recording start notification. This eliminates the need to notify the video conference application 710 every time a recording start notification is received.

[0307] S6, S7: When the video conference application 710 determines that it has received a recording start notification for the first time, the interface unit 730a makes a UI change request to the UI image generation unit 33 via the interface processing unit 720a. The UI change request is a request to display the operating status of the transmission terminal 10dd on the electronic whiteboard UI screen 740. Specifically, it is displayed that the transmission terminal 10dd is recording.

[0308] S8: As a result, the UI image generating unit 33 of the electronic whiteboard application 720 displays the electronic whiteboard UI screen 740 indicating that the transmission terminal 10dd is recording. An example of the electronic whiteboard UI screen 740 will be described with reference to FIG.

[0309] At this point, the interface unit 730a does not need to transmit a videoconference UI screen switching signal to the OS 700. First, in a situation where the electronic whiteboard UI screen 740 is displayed on the display 3 of the electronic whiteboard 8, a message indicating that the transmission terminal 10dd is recording is displayed in step S8, so that the user can notice that recording is being performed at another location. In a situation where the videoconference UI screen 750 is displayed on the display 3 of the electronic whiteboard 8, the recording location mark 71 or the other location recording mark 72 described in the first embodiment is displayed on the full screen display screen 501 or the information display screen 601, so that the user can notice that recording is being performed at another location.

[0310] Similarly, the interface unit 730a can detect that the video conference application 710 has received a recording end notification. In this case, the interface unit 730a makes a UI change request to the UI image generating unit 33 of the electronic whiteboard application 720. This UI change request is a request to erase the information indicating that the transmission terminal 10dd is recording.

[0311] Fig. 34 is an example of an electronic whiteboard UI screen 740 that displays the fact that another location is recording. In Fig. 34, text 761 saying "Recording conference" is displayed in a ticker section 760. The ticker section 760 is an area provided in a part of the screen where information unrelated to the content, such as an image, is displayed. Generally, this area is created by shrinking the content portion while maintaining the aspect ratio of the content portion. In this way, it is possible to display the fact that another location is recording without overlapping with the content.

[0312] However, the interactive whiteboard UI screen 740 does not have to be particularly reduced in size in FIG. 34, and may be displayed, for example, superimposed on the four corners or center of the content (handwritten strokes, etc.) of the interactive whiteboard UI screen 740.

[0313] 34, the UI image generating unit 33 may display the fact that recording is in progress by blinking the video conference icon 702a, changing the color, etc. Also, instead of displaying the fact that recording is in progress using text as in Fig. 34, an icon indicating that recording is in progress may be displayed instead of the text 761.

[0314] <Change display order> Next, a procedure for changing the display order of the electronic whiteboard UI screen 740 and the video conference UI screen 750 will be described with reference to Fig. 35. Fig. 35 is an example of a sequence diagram showing a procedure for the interface unit 730a to change the display order of the electronic whiteboard UI screen 740 and the video conference UI screen 750. Fig. 35 is executed while a video conference is being held. The description will be given assuming that the video conference UI screen 750 is displayed on the display 3 of the electronic whiteboard 8.

[0315] S1: The user presses the electronic whiteboard icon 703 on the video conference UI screen 750. The operation input receiving unit 62 receives this operation and notifies the interface processing unit 710a.

[0316] S2: Upon receiving this notification, the interface processor 710a transmits an electronic whiteboard UI screen switching signal to the interface unit 730a.

[0317] S3: The interface unit 730a transmits an electronic whiteboard UI screen switching signal to the OS 700.

[0318] S4: The OS 700 displays the electronic whiteboard UI screen 740 generated by the image superimposition unit 28 of the electronic whiteboard application 720 on the front of the display 3. The OS 700 also activates the electronic whiteboard UI screen 740. That is, the electronic whiteboard application 720 that created the electronic whiteboard UI screen 740 is made the target of the user's operations. The OS 700 also assigns a slightly higher priority to the electronic whiteboard application 720 than other applications. As a result, the electronic whiteboard UI screen 740 is displayed on the front of the display 3 while the video conference application 710 is also operating.

[0319] In addition, when the interactive whiteboard application 720 is displayed in the foreground, in order to display the video conference application 710 in the foreground, the procedure in FIG. 35 can be executed in reverse.

[0320] In this way, the user can switch between the electronic whiteboard application 720 and the video conference application 710 at will.

[0321] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0322] For example, the relay device 30, the transmission management device 50, and the program providing device 90 may be constructed by a single computer, rather than being independent of each other. Also, each means and the storage units 1000, 2000, and 5000 of the transmission terminal 10, the transmission management device 50, and the program providing device 90 may be arbitrarily assigned to a plurality of computers. In this case, the UI providing device 20 may be included as a function of any of the computers.

[0323] In the above embodiment, a video conference system has been described as an example of the transmission system 1, but the present invention is not limited to this and may be a telephone system such as an IP (Internet Protocol) telephone or an Internet telephone.

[0324] Furthermore, each transmission terminal 10 may communicate without going through the transmission management device 50. WebRTC is known as an example of such a communication protocol.

[0325] The transmission system 1 may be a system including a car navigation device mounted on a moving object. A PC for screen sharing may be connected to the transmission terminal 10.

[0326] In the above embodiment, image data and audio data are described as an example of content data, but the present invention is not limited to this and may be tactile data. In this case, the sensation of a user touching one transmission terminal is transmitted to the other transmission terminal.

[0327] Furthermore, the content data may be smell data. In this case, the smell (odor) of one transmission terminal side is transmitted to the other transmission terminal side. The content data may be at least one of image data, audio data, tactile data, and smell data.

[0328] In addition, in the above embodiment, a case where a video conference is held using the transmission system 1 has been described, but this is not limited to this, and the system may also be used for meetings, general conversations between family members or friends, etc., or for the presentation of information in one direction.

[0329] In addition, in the above embodiment, participants were mainly notified that image data was being recorded, but participants may also be notified that audio data is being recorded when only audio data is being transmitted and received.

[0330] The transmitting / receiving unit 11 is an example of a receiving means, the base list display unit 18b is an example of a recording terminal notifying means, the recording management table 1000b is an example of an identification information storage means, the full screen display screen 501 is an example of a first screen, and the information display screen 601 is an example of a second screen. The transmission management device 50 is an example of an external device, the destination list display unit 16 is an example of a destination list display means, and the display 120 is an example of a display device. The display control unit 17 is an example of a display control means, the operation input receiving unit 12 is an example of a selection receiving means, and the auxiliary area display unit 18a is an example of a recording notifying means. The destination list management DB 5003 is an example of a destination list storage means, the terminal management DB 5002 is an example of a terminal state storage means, the image recording unit 41 is an example of a recording means, and the transmitting / receiving unit 51 is an example of a transmitting means. The transmission terminal 10 is an example of a first information processing device (or a second information processing device), and the transmission management device 50 is an example of a second information processing device. The display control unit 66 is an example of a first display control means. The UI image generating unit 33 and the video superimposing unit 28 are an example of a second display control means, and the interface unit 703a is an example of a notification means. [Explanation of symbols]

[0331] 1. Transmission System 10 Transmission Terminal 12 Operation input reception section 17 Display control section 18a Auxiliary area display section 18b Location list display section 50 Transmission management device 52 Terminal authentication unit 53 Status Management Unit 54 Terminal Extraction Unit 55 Terminal status acquisition unit 120 Display [Prior art documents] [Patent documents]

[0332] [Patent Document 1] JP 2011-216984 A

Claims

1. An information processing device connected to one or more other information processing devices via a network, a reception means for receiving a request to participate in a conference with the one or more other information processing devices; a display control means for causing a display device to display a first screen displaying image data transmitted from the one or more other information processing devices while the user is participating in a conference related to the request received by the receiving means, The display control means when one of the one or more other information processing devices starts recording image data of the conference while the first screen is being displayed, first recording notification information indicating that the image data of the conference is being recorded is displayed on the first screen; when a request to display information related to the one or more other information processing devices on the display device is received while the first recording notification information is being displayed on the first screen, second recording notification information indicating that one of the one or more other information processing devices is recording image data of the conference is displayed on a second screen displaying information related to the one or more other information processing devices.

23. An information processing apparatus comprising:

2. An information processing device as described in Claim 1, wherein the display control means displays on the second screen for each of the one or more other information processing devices that the one information processing device is recording image data or audio data.

3. When any one of the one or more other information processing devices starts recording image data of the conference, 3. The information processing apparatus according to claim 2, wherein said display control means causes said first recording notification information to be constantly displayed on said first screen.

4. When any one of the one or more other information processing devices starts recording image data of the conference, 4. The information processing apparatus according to claim 3, wherein the display control means displays the second recording notification information on the second screen.

5. When the one of the one or more other information processing devices starts recording image data of the conference while the first recording notification information is not displayed on the first screen, the display control means starts displaying the first recording notification information on the first screen, When the one of the one or more other information processing devices starts recording image data of the conference while the second recording notification information is not displayed on the second screen, 5. The information processing apparatus according to claim 4, wherein the display control means starts displaying the second recording notification information.

6. having an identification information storage means for storing identification information of said one information processing device, When a recording end notification indicating that the one information processing device is to end recording of the image data or audio data transmitted by the one information processing device is received together with identification information of the one information processing device, the identification information of the one information processing device that has transmitted the recording end notification is deleted from the identification information storage means, and no identification information is stored in the identification information storage means; The display control means causes the first recording notification information on the first screen to be hidden, 6. The information processing apparatus according to claim 4, wherein the second recording notification information indicating that the one information processing apparatus is recording image data of the conference is not displayed on the second screen.

7. When the receiving means receives, from an external device that monitors the alive status of the one information processing device, together with identification information of the one information processing device, information on a change in operation state indicating that communication has been disconnected, the identification information of the one information processing device transmitted by the external device is deleted from the identification information storage means, and when no identification information is stored in the identification information storage means, the display control means makes the first recording notification information on the first screen non-display; 7. The information processing apparatus according to claim 6, wherein the display control means makes the second recording notification information, which indicates that the one information processing apparatus is recording image data of the conference, invisible on the second screen.

8. from an external device that has a destination list of the one or more other information processing devices with which the information processing device can start communication, and transfers the first record notification information and the identification information of the one information processing device transmitted by the one information processing device to the information processing device; the receiving means receives information regarding an operating state of each of the information processing devices in the destination list, the information including whether image data of the conference is being recorded; The information processing device according to any one of claims 1 to 7, further comprising a destination list display means for displaying on a display device whether or not one of the information processing devices in the destination list is recording image data of the conference based on information regarding the operating status.

9. a selection receiving means for receiving a selection of the one or more other information processing devices to be communication destinations; The information processing device according to claim 8, wherein when the selection accepting means accepts the selection of the one information processing device that is currently recording image data of the conference, the destination list display means displays on a display device an indication that communication will be performed with the one information processing device that is currently recording image data of the conference.

10. A display method performed by an information processing device connected to one or more other information processing devices via a network, comprising: an acceptance process for accepting a request to participate in a conference with the one or more other information processing devices; a process of displaying, on a display device, a first screen that displays image data transmitted from the one or more other information processing devices while participating in the conference related to the request accepted in the acceptance process; when one of the one or more other information processing devices starts recording image data of the conference while the first screen is being displayed, first recording notification information indicating that the image data of the conference is being recorded is displayed on the first screen; when a request to display information related to the one or more other information processing devices on the display device is received while the first recording notification information is being displayed on the first screen, a process of displaying second recording notification information indicating that one of the one or more other information processing devices is recording image data of the conference on a second screen displaying information related to the one or more other information processing devices; A display method comprising:

11. A display system having one or more other information processing devices and an information processing device connected to the one or more other information processing devices via a network, The information processing device includes: a reception means for receiving a request to participate in a conference with the one or more other information processing devices; a display control means for causing a display device to display a first screen displaying image data transmitted from the one or more other information processing devices while the user is participating in a conference related to the request received by the receiving means, The display control means when one of the one or more other information processing devices starts recording image data of the conference while the first screen is being displayed, first recording notification information indicating that the image data of the conference is being recorded is displayed on the first screen; when a request to display information related to the one or more other information processing devices on the display device is received while the first recording notification information is being displayed on the first screen, second recording notification information indicating that one of the one or more other information processing devices is recording image data of the conference is displayed on a second screen displaying information related to the one or more other information processing devices. A display system comprising:

12. An information processing device connected to one or more other information processing devices via a network, a reception means for receiving a request to participate in a conference with the one or more other information processing devices; a display control unit that causes a display device to display a first screen displaying image data transmitted from the one or more other information processing devices when the user is participating in a conference related to the request received by the reception unit, The display control means when one of the one or more other information processing devices starts recording image data of the conference while the first screen is being displayed, first recording notification information indicating that the image data of the conference is being recorded is displayed on the first screen; when a request to display information related to the one or more other information processing devices on the display device is received while the first recording notification information is being displayed on the first screen, second recording notification information indicating that one of the one or more other information processing devices is recording image data of the conference is displayed on a second screen displaying information related to the one or more other information processing devices. A program characterized by:

13. The program described in Claim 12, wherein the display control means displays on the second screen for each of the one or more other information processing devices that the one information processing device is recording image data or audio data.

14. When any one of the one or more other information processing devices starts recording image data of the conference, 14. The program according to claim 13, wherein the display control means causes the first recording notification information to be constantly displayed on the first screen.

15. When any one of the one or more other information processing devices starts recording image data of the conference, 15. The program according to claim 14, wherein the display control means displays the second recording notification information on the second screen.

16. When one of the one or more other information processing devices starts recording image data of the conference while the first recording notification information is not displayed on the first screen, the display control means starts displaying the first recording notification information on the first screen, When the one of the one or more other information processing devices starts recording image data of the conference while the second recording notification information is not displayed on the second screen, 15. The program according to claim 14, wherein the display control means starts displaying the second recording notification information.

17. The information processing device has an identification information storage means for storing identification information of the one information processing device, When a recording end notification indicating that the one information processing device is to end recording of the image data or audio data transmitted by the one information processing device is received together with identification information of the one information processing device, the identification information of the one information processing device that has transmitted the recording end notification is deleted from the identification information storage means, and no identification information is stored in the identification information storage means; The display control means causes the first recording notification information on the first screen to be hidden, 17. The program according to claim 15, further comprising: making the second recording notification information, which indicates that the one information processing apparatus is recording image data of the conference, invisible on the second screen.

18. When the receiving means receives, from an external device that monitors the alive state of the one information processing device, together with identification information of the one information processing device, information on a change in operation state indicating that communication has been disconnected, the identification information of the one information processing device transmitted by the external device is deleted from the identification information storage means, and when no identification information is stored in the identification information storage means, the display control means makes the first recording notification information on the first screen non-display; 18. The program according to claim 17, wherein the display control means makes the second recording notification information, which indicates that the one information processing device is recording image data of the conference, invisible on the second screen.

19. from an external device which has a destination list of the one or more other information processing devices with which the information processing device can initiate communication, and which transfers the first record notification information transmitted by the one information processing device and identification information of the one information processing device to the one information processing device; the receiving means receives information regarding an operating state of each of the information processing devices in the destination list, the information including whether image data of the conference is being recorded; The information processing device, The program according to any one of claims 12 to 18, which functions as a destination list display means for displaying on a display device whether or not the one information processing device in the destination list is recording, based on the information regarding the operating status.

20. The information processing device, functioning as a selection receiving means for receiving a selection of the one or more other information processing devices to be communication destinations; The program described in claim 19, wherein when the selection accepting means accepts the selection of the one information processing device that is currently recording image data of the conference, the destination list display means displays on a display device an indication that communication will be performed with the one information processing device that is currently recording image data of the conference.

Images